Posts Tagged ‘Microsoft’

Keeping your events thread safe

March 11, 2012

An area I’ve noticed where engineers often forget to think about synchronization is where firing events.
Now I’m going to go over a little background on C# delegates quickly just to refresh what we learnt or should have learnt years ago at the beginnings of the C# language.

It seems to be a common misconception, that all that is needed to keep synchronisation,
is to check the delegate (technically a MulticastDelegate, or in architectural terms the publisher of the publish-subscribe pattern (more commonly known as the observer pattern)) for null.

Defining the publisher without using the event keyword

public class Publisher {
   // ...

   // Define the delegate data type
   public delegate void MyDelegateType();

   // Define the event publisher
   public MyDelegateType OnStateChange {
      get{ return _onStateChange;}
      set{ _onStateChange = value;}
   }
   private MyDelegateType _onStateChange;

   // ...
}

When you declare a delegate, you are actually declaring a MulticastDelegate.
The delegate keyword is an alias for a type derived from System.MulticastDelegate.
When you create a delegate, the compiler automatically employs the System.MulticastDelegate type rather than the System.Delegate type.
When you add a method to a multicast delegate, the MulticastDelegate class creates a new instance of the delegate type, stores the object reference and the method pointer for the added method into the new instance, and adds the new delegate instance as the next item in a list of delegate instances.
Essentially, the MulticastDelegate keeps a linked list of Delegate objects.

It’s possible to assign new subscribers to delegate instances, replacing existing subscribers with new subscribers by using the = operator.
Most of the time what is intended is actually the += operator (implemented internally by using System.Delegate.Combine()).
System.Delegate.Remove() is what’s used when you use the -+ operator on a delegate.

class Program {
   public static void Main() {

      Publisher publisher = new Publisher();
      Subscriber1 subscriber1 = new Subscriber1();
      Subscripber2 subscripber2 = new Subscripber2();

      publisher.OnStateChange = subscriber1.OnStateChanged;

      // Bug: assignment operator overrides previous assignment.
      // if using the event keyword, the assignment operator is not supported for objects outside of the containing class.
      publisher.OnStateChange = subscriber2.OnStateChanged;

   }
}

Another short coming of the delegate is that delegate instances are able to be invoked outside of the containing class.

class Program {
   public static void Main() {
      Publisher publisher = new Publisher();
      Subscriber1 subscriber1 = new Subscriber1();
      Subscriber2 subscriber2 = new Subscriber2();

      publisher.OnStateChange += subscriber1.OnStateChanged;
      publisher.OnStateChange += subscriber2.OnStateChanged;

      // lack of encapsulation
      publisher.OnStateChange();
   }
}

C# Events come to the rescue

in the form of the event keyword.
The event keyword address’s the above problems.

The modified Publisher looks like the following:

public class Publisher {
   // ...

   // Define the delegate data type
   public delegate void MyDelegateType();

   // Define the event publisher
   public event MyDelegateType OnStateChange;

   // ...
}

Now. On to synchronisation

The following is an example from the GoF guys with some small modifications I added.
You’ll also notice, that the above inadequacies are taken care of.
Now if the Stock.OnChange is not accessed by multiple threads, this code is fine.
If it is accessed by multiple threads, it’s not fine.
Why I hear you ask?
Well, between the time the null check is performed on the Change event
and when Change is fired, Change could be set to null, by another thread.
This will of course produce a NullReferenceException.

The code on lines 59,60 is not atomic.

using System;
using System.Collections.Generic;

namespace DoFactory.GangOfFour.Observer.NETOptimized {
    /// <summary>
    /// MainApp startup class for .NET optimized
    /// Observer Design Pattern.
    /// </summary>
    class MainApp {
        /// <summary>
        /// Entry point into console application.
        /// </summary>
        static void Main() {
            // Create IBM stock and attach investors
            var ibm = new IBM(120.00);

            // Attach 'listeners', i.e. Investors
            ibm.Attach(new Investor { Name = "Sorros" });
            ibm.Attach(new Investor { Name = "Berkshire" });

            // Fluctuating prices will notify listening investors
            ibm.Price = 120.10;
            ibm.Price = 121.00;
            ibm.Price = 120.50;
            ibm.Price = 120.75;

            // Wait for user
            Console.ReadKey();
        }
    }

    // Custom event arguments
    public class ChangeEventArgs : EventArgs {
        // Gets or sets symbol
        public string Symbol { get; set; }

        // Gets or sets price
        public double Price { get; set; }
    }

    /// <summary>
    /// The 'Subject' abstract class
    /// </summary>
    abstract class Stock {
        protected string _symbol;
        protected double _price;

        // Constructor
        public Stock(string symbol, double price) {
            this._symbol = symbol;
            this._price = price;
        }

        // Event
        public event EventHandler<ChangeEventArgs> Change;

        // Invoke the Change event
        private void OnChange(ChangeEventArgs e) {
            // not thread safe
            if (Change != null)
                Change(this, e);
        }

        public void Attach(IInvestor investor) {
            Change += investor.Update;
        }

        public void Detach(IInvestor investor) {
            Change -= investor.Update;
        }

        // Gets or sets the price
        public double Price {
            get { return _price; }
            set {
                if (_price != value) {
                    _price = value;
                    OnChange(new ChangeEventArgs { Symbol = _symbol, Price = _price });
                    Console.WriteLine("");
                }
            }
        }
    }

    /// <summary>
    /// The 'ConcreteSubject' class
    /// </summary>
    class IBM : Stock {
        // Constructor - symbol for IBM is always same
        public IBM(double price)
            : base("IBM", price) {
        }
    }

    /// <summary>
    /// The 'Observer' interface
    /// </summary>
    interface IInvestor {
        void Update(object sender, ChangeEventArgs e);
    }

    /// <summary>
    /// The 'ConcreteObserver' class
    /// </summary>
    class Investor : IInvestor {
        // Gets or sets the investor name
        public string Name { get; set; }

        // Gets or sets the stock
        public Stock Stock { get; set; }

        public void Update(object sender, ChangeEventArgs e) {
            Console.WriteLine("Notified {0} of {1}'s " +
                "change to {2:C}", Name, e.Symbol, e.Price);
        }
    }
}

At least we don’t have to worry about the += and -= operators. They are thread safe.

Ok. So how do we make it thread safe?
Now I’ll do my best not to make your brain hurt.
We can assign a local copy of the event and then check that instead.
How does that work you say?
The Change delegate is a reference type.
You may think that  threadSafeChange references the same location as Change,
thus any changes to Change would also be reflected in threadSafeChange.
That’s not the case though.
Change += investor.Update does not add a new delegate to Change, but assigns it a new MulticastDelegate,
which has no effect on the original MulticastDelegate that threadSafeChange also references.

The reference part of reference type local variables is stored on the stack.
A new stack frame is created for each thread with every method call
(whether its an instance or static method).
All local variables are safe…
so long as they are not reference types being passed to another thread or being passed to another thread by ref.
So, only one thread can access the threadSafeChange instance.

private void OnChange(ChangeEventArgs e) {
   // assign reference to heap allocated memory to stack allocated implements thread safety
   EventHandler<ChangeEventArgs> threadSafeChange = Change;
   if ( threadSafeChange != null)
      threadSafeChange(this, e);
}

Now for a bit of error handling

If one subscriber throws an exception, any subscribers later in the chain do not receive the publication.
One way to get around this problem, is to semantically override the enumeration of the subscribers.
Thus providing the error handling.

private void OnChange(ChangeEventArgs e) {
   // assign reference to heap allocated memory to stack allocated implements thread safety
   EventHandler<ChangeEventArgs> threadSafeChange = Change;
   if ( threadSafeChange != null) {
      foreach(EventHandler<ChangeEventArgs> handler in Change.GetInvocationList()) {
         try {
            //if subscribers delegate methods throw an exception, we'll handle in the catch and carry on with the next delegate
            handler(this, e);
            // if we only want to allow a single subscriber
            if (Change.GetInvocationList().Length > 1)
               throw new Exception("Too many subscriptions to the Stock.Change" /*, provide a meaningful inner exception*/);
         }
         catch (Exception exception) {
            // what we do here depends on what stage of development we are in.
            // if we're in early stages, pre-release, fail early and hard.
         }
      }
   }
}

Quick walk through, of my UPS library

August 4, 2011

Part three of a three part series

On setting up a UPS solution, to enable clean shutdown of vital network components.

In this post, we’ll be reviewing the library that performs the shutting down of our servers.

When I started on the library PowerOffUPSGuests.dll,
my thoughts were, if I’m going to do this, I wanted it to be extensible.
Able to shutdown pretty much any machines, requiring a clean shutdown due to power failure.
What I’ve done is left points to be easily extended in the future, when new requirements present themselves.

Source Code

The PowerOffUPSGuests repository is on bitbucket

I’m assuming you know how to use Mercurial and have it installed on your dev machine.
If you’re not familiar with Mercurial (hg) There’s a little here to get your feet wet.
Besides that, there is plenty of very good documentation on the net.
For starters, you’ll need to create a directory that you want to have as your repository.
For example, I use C:\Scripts.
Set this directory as a repository.
Then from within the directory,
issue an hg pull https://bitbucket.org/LethalDuck/poweroffupsguests
Then update your working directory to the tip of the local repository.

You’ll need a BinaryMist.PowerOffUPSGuests.dll.config in your <repository>\UPS\PowerOffUPSGuests\PowerOffUPSGuests\
which should look something like the following.
At this stage I’ve only been shutting down an ESXi host.
Replace the value at line 22 with the user that has privileges to perform shutdown on the server.
Replace the value at line 23 with the absolute path to the password file you’re about to generate.
Line 28 is the class name of the ServerController.
Line 34 denotes whether or not the Initiator will perform the shutdowns synchronously or asynchronously
The Server[n] and ServerPort[n] values that are commented out, are used when you want to intercept the messages being sent to/from the target server.
This is useful for examination and to help build the appropriate messages that the target server expects.

<?xml version="1.0"?>
<configuration>
   <assemblySettings>
      <!--As aditional target servers are added to the queue to be shutdown
         Keep the same nameing convention used below
         Just increment the suffix number for each target servers key.
         The first target suffix must start at 0.
         Additional target suffix's must be sequential.
         -->

      <!--Target servers to be shutdown-->

      <!--FreeNAS-->
      <!--add key="ServerUser0" value="YourUser"/>
      <add key="ServerUserPwFile0" value="Absolute directory that your password file resides\FileServerPw"/-->
      <!--add key="Server0" value="127.0.0.1"/--><!--localhost used for interception-->
      <!--add key="Server0" value="YourFileServerName"/-->
      <!--add key="ServerPort0" value="8080"/--><!--port used for interception-->
      <!--add key="ServerPort0" value="443"/-->
      <!--add key="Controller0" value="FreeNASController"/-->

      <!--ESXi-->
      <add key="ServerUser0" value="YourUser"/>
      <add key="ServerUserPwFile0" value="Absolute directory that your password file resides\VMHostPw"/>
      <!--add key="Server1" value="127.0.0.1"/--><!--localhost used for interception-->
      <add key="Server0" value="YourVSphereHostName"/>
      <!--add key="ServerPort1" value="8080"/--><!--port used for interception-->
      <add key="ServerPort0" value="443"/>
      <add key="Controller0" value="VMServerController"/>

      <!--Assembly settings-->

      <add key="LogFilePath" value="Some absolute path\Log.txt"/>
      <add key="CredentialEntropy" value="A set of comma seperated digits"/><!--"4,2,7,9,1" for example-->
      <add key="Synchronicity" value="Synchronous"/> <!--Check other values in Initiator.Synchronicity-->
      <add key="IgnoreSslErrors" value="true"/>
      <add key="Debug" value="true"/>

   </assemblySettings>
<startup><supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.0"/></startup></configuration>

Build the solution

You’ll now need to navigate to <repository>\UPS\PowerOffUPSGuests\
and run the file PowerOffUPSGuests.sln.
Build.
You should now notice a couple of binaries in <repository>\UPS\
along with the libraries config file.
As I mentioned in part two, the PCNS will execute <repository>\UPS\PowerOff.bat which will run PowerOffUPSGuests.ps1.
Which will inturn kick off the BinaryMist.PowerOffUPSGuests.dll which does the work.

Generate the encrypted password file

Just run the BinaryMist.PasswordFileCreator.exe.
This will provide the required user prompts to capture the password for the vSphere host you’re intending to shutdown.
Or if you would like to extend the project and create a specialized ServerController for your needs.
You can use the BinaryMist.PasswordFileCreator to capture any credentials and save to file.

The code that performs the capture and encryption looks like the following:


        static void Main() {
            CreatePasswordFile();
        }

        /// <summary>
        /// Provides interactive capture for the insertion of an encrypted password,
        /// based on the ServerUserPwFile0 specified in the BinaryMist.PowerOffUPSGuests.dll.config file.
        /// </summary>
        public static void CreatePasswordFile() {

            bool validPath;
            string path = null;
            string RetryMessage = "Please try again.";

            Console.WriteLine("You must create the password file running under" + Initiator.NewLine + "the same account that will run BinaryMist.PowerOffUPSGuests.");

            do {
                Console.WriteLine(
                    "From the BinaryMist.PowerOffUPSGuests.dll.config file." + Initiator.NewLine +
                    "Please specify the ServerUserPwFile[n] value" + Initiator.NewLine +
                    "for the encrypted Password to be stored to." + Initiator.NewLine +
                    "This must be a valid path" + Initiator.NewLine
                );

                try {
                    validPath = true;
                    path = Path.GetFullPath(Console.ReadLine());

                    if (!
                        ((IEnumerable<string>)ConfigReader.Read.AllKeyVals.Values)
                        .Contains<string>(
                            path, StringComparer.CurrentCultureIgnoreCase
                        )
                    ) {
                        Console.WriteLine(Initiator.NewLine);
                        Console.WriteLine("The value that was entered" + Initiator.NewLine +
                            "was not one of the specified values for ServerUserPwFile[n]");
                        Console.WriteLine(RetryMessage + Initiator.NewLine);
                        validPath = false;
                    }
                } catch (Exception) {
                    Console.WriteLine(Initiator.NewLine + "An invalid path was entered." + Initiator.NewLine + RetryMessage + Initiator.NewLine);
                    validPath = false;
                }
            } while (validPath == false);

            Console.WriteLine(
                Initiator.NewLine
                + "The password you are about to enter"
                + Initiator.NewLine
                + "will be encrypted to file \"{0}\""
                , path
            );

            byte[] encryptedBytes = ProtectedData.Protect(
                new ASCIIEncoding().GetBytes(pWord),
                ServerAdminDetails.CredentialEntropy(),
                DataProtectionScope.CurrentUser
            );
            File.WriteAllBytes(path, encryptedBytes);

            Console.WriteLine(
                Initiator.NewLine
                + Initiator.NewLine
                + string.Format(
                    "The password you just entered has been encrypted"
                    + Initiator.NewLine
                    + "and saved to {0}"
                    , path
                )
            );
            Console.WriteLine(Initiator.NewLine + "Press any key to exit");
            Console.ReadKey(true);
        }

 


        private static string pWord {
            get {
                bool passWordsMatch = false;
                bool firstAttempt = true;
                string passWord = null;
                while (!passWordsMatch) {

                    if (!firstAttempt)
                        Console.WriteLine(Initiator.NewLine + "The passwords did not match." + Initiator.NewLine);

                    Console.WriteLine(Initiator.NewLine + "Please enter the password..." + Initiator.NewLine);
                    string passWordFirstAttempt = GetPWordFromUser();
                    Console.WriteLine(Initiator.NewLine + Initiator.NewLine + "Please confirm by entering the password once again..." + Initiator.NewLine);
                    string passWordSecondAttempt = GetPWordFromUser();

                    if (string.Compare(passWordFirstAttempt, passWordSecondAttempt) == 0) {
                        passWordsMatch = true;
                        passWord = passWordFirstAttempt;
                    }
                    firstAttempt = false;
                }
                Console.WriteLine(Initiator.NewLine + Initiator.NewLine + "Success, the passwords match." + Initiator.NewLine);
                return passWord;
            }
        }

 


        private static string GetPWordFromUser() {
            string passWord = string.Empty;
            ConsoleKeyInfo info = Console.ReadKey(true);
            while (info.Key != ConsoleKey.Enter) {
                if (info.Key != ConsoleKey.Backspace) {
                    if (info.KeyChar < 0x20 || info.KeyChar > 0x7E) {
                        info = Console.ReadKey(true);
                        continue;
                    }
                    passWord += info.KeyChar;
                    Console.Write("*");
                    info = Console.ReadKey(true);
                } else if (info.Key == ConsoleKey.Backspace) {
                    if (!string.IsNullOrEmpty(passWord)) {
                        passWord = passWord.Substring
                            (0, passWord.Length - 1);
                        Console.SetCursorPosition(Console.CursorLeft - 1, Console.CursorTop);
                        Console.Write(' ');
                        Console.SetCursorPosition(Console.CursorLeft - 1, Console.CursorTop);
                    }
                    info = Console.ReadKey(true);
                }
            }
            return passWord;
        }

Once you’ve created the password file, you’re pretty much ready to start testing.
If you’ve followed the directions covered in the first two parts of this series, you should be good to go.
Part one.
Part two.

We’ll quickly go through some of the more interesting parts of the code…

The ConfigReader

The constructor loads the _settings IDictionary by calling the ReadConfig function.
.net libraries don’t usually contain an assembly config file.
This is how we get around it.
We read the name of the assemblies config file (see line 70).
We then load the configuration into an XmlDocument.
Create and populate the XmlNodeList.
Return the populated IDictionary.

    /// <summary>
    /// Reads the libraries configuration into memory.
    /// Provides convienient readonly access of the configuration from a single instance.
    /// </summary>
    public class ConfigReader {

        #region singleton initialization
        private static readonly Lazy<ConfigReader> _instance = new Lazy<ConfigReader>(() => new ConfigReader());

        /// <summary>
        /// constructor that sets the value of our "_settings" variable
        /// </summary>
        private ConfigReader() {
            _settings = ReadConfig(Assembly.GetCallingAssembly());
        }

        /// <summary>
        /// The first time Read is called must be from within this assembly,
        /// in order to create the instance of this class for the containing assembly.
        /// </summary>
        public static ConfigReader Read {
            get {
                return _instance.Value;
            }
        }
        #endregion

        /// <summary>
        /// settings to be used throughout the class
        /// </summary>
        private IDictionary _settings;
        /// <summary>
        /// constant name for the node name we're looking for
        /// </summary>
        private const string NodeName = "assemblySettings";

        /// <summary>
        /// class Indexer.
        /// Provides the value of the specified key in the config file.
        /// If the key doesn't exist, an empty string is returned.
        /// </summary>
        public string this[string key] {
            get {
                string settingValue = null;

                if (_settings != null) {
                    settingValue = _settings[key] as string;
                }

                return settingValue ?? string.Empty;
            }
        }

        public IDictionary<string, string> AllKeyVals {
            get {
                IDictionary<string, string> settings = new Dictionary<string, string>();
                foreach(DictionaryEntry item in _settings) {
                    settings.Add((string)item.Key, (string)item.Value);
                }
                return settings;
            }
        }

        /// <summary>
        /// Open and parse the config file for the provided assembly
        /// </summary>
        /// <param name="assembly">The assembly that has a config file.</param>
        /// <returns></returns>
        private static IDictionary ReadConfig(Assembly assembly) {
            try {
                string cfgFile = assembly.CodeBase + ".config";

                XmlDocument doc = new XmlDocument();
                doc.Load(new XmlTextReader(cfgFile));
                XmlNodeList nodes = doc.GetElementsByTagName(NodeName);

                foreach (XmlNode node in nodes) {
                    if (node.LocalName == NodeName) {
                        DictionarySectionHandler handler = new DictionarySectionHandler();
                        return (IDictionary)handler.Create(null, null, node);
                    }
                }
            } catch (Exception e) {
                Logger.Instance.Log(e.Message);
            }

            return (null);
        }

        #region Config related settings

        /// <summary>
        /// Readonly value, specifying whether debug is set to true in the assemblySettings of the BinaryMist.PowerOffUPSGuests.dll.config file.
        /// </summary>
        public bool Debug {
            get {
                if (_debug != null)
                    return _debug == true ? true : false;
                _debug = Read["Debug"] == "true" ? true : false;
                return _debug == true ? true : false;
            }
        }
        private bool? _debug;

        #endregion

    }

The Initiator


private void ShutdownSynchronously(Queue<ServerController> serverControllers) {
    foreach (ServerController serverController in serverControllers) {
        serverController.Shutdown();
    }
}

private void ShutdownAsynchronously(Queue<ServerController> serverControllers) {
    Action[] shutdownActions = new Action[serverControllers.Count];
    ServerController[] serverControllerArray = serverControllers.ToArray();

    for (int i = 0; i < serverControllerArray.Length; i++) {
        shutdownActions[i] = serverControllerArray[i].Shutdown;
    }

    try {
        Parallel.Invoke(shutdownActions);
    }
        // No exception is expected in this example, but if one is still thrown from a task,
        // it will be wrapped in AggregateException and propagated to the main thread. See MSDN example
    catch (AggregateException e) {
        Logger.Instance.Log(string.Format("An action has thrown an exception. THIS WAS UNEXPECTED.\n{0}", e.InnerException));
        throw new Exception();
    }
}

public string InitShutdownOfServers() {
    Logger.Instance.LogTrace();
    Queue<ServerController> serverControllers = new Queue<ServerController>();

    try {
        foreach (ServerAdminDetails serverAdminDetail in ServerAdminDetails.QueuedDetails) {

            Type t = Type.GetType(GetType().Namespace + "." + serverAdminDetail.ServerControllerType);
            serverControllers.Enqueue(Activator.CreateInstance(t, serverAdminDetail) as ServerController);
        }
    } catch(Exception e) {
        Logger.Instance.Log("Exception occured while enqueueing the server controllers. Details follow:" +
            NewLine +
            e.ToString()
        );
        throw;
    }

    bool ignoreCase = true;
    Synchronicity synchronicity = (Synchronicity)Enum.Parse(typeof (Synchronicity), ConfigReader.Read["Synchronicity"], ignoreCase);

    if(synchronicity == Synchronicity.Synchronous)
        ShutdownSynchronously(serverControllers);
    else
        ShutdownAsynchronously(serverControllers);

    return "InitShutdownOfServers successfully executed.";
}

In order to place the ServerController‘s on the serverController‘s Queue (line 33)… via the iterator provided by the Queue of ServerAdminDetails returned from
the static QueuedDetails property of ServerAdminDetails,
We must first instantiate the single instance of ServerAdminDetails,
which is what line 30 does.

In order for ServerAdminDetails to be constructed, the static _queue member (which is a Lazy of Queue of ServerAdminDetails) must be initialized first.
In order for the _queue member to be initialized with a Queue of ServerAdminDetails, the static QueueServersForShutdown procedure must be called.

This is where we pull out the values from the BinaryMist.PowerOffUPSGuests.dll.config shown above with the help of the ConfigReader.
As you can see, we iterate through the config file, building up the Queue of ServerAdminDetails  until we’ve read all the appropriate values.
Each pass through the loop instantiates a new ServerAdminDetails (non singleton because of inside class scope) with the values we pulled from the config file.

The ServerAdminDetails

    /// <summary>
    /// Provides the administration details of the servers listed in the BinaryMist.PowerOffUPSGuests.dll.config file.
    /// </summary>
    public class ServerAdminDetails {

        #region singleton initialization
        private static readonly Lazy<Queue<ServerAdminDetails>> _queue = new Lazy<Queue<ServerAdminDetails>>(QueueServersForShutdown);

        private ServerAdminDetails(string serverController, string serverName, string serverPort, string userName, byte[] serverCredential) {
            ServerControllerType = serverController;
            ServerName = serverName;
            ServerPort = serverPort;
            UserName = userName;
            Password = serverCredential;
        }

        /// <summary>
        /// Provides the process wide single instance queue of each servers admin details.
        /// </summary>
        public static Queue<ServerAdminDetails> QueuedDetails {
            get {
                return _queue.Value;
            }
        }
        #endregion

        private static Queue<ServerAdminDetails> ServersQueuedForShutdown { get; set; }

        private static byte[] GetMyCredential(string pWFileName) {
            try {
                return File.ReadAllBytes(pWFileName);
            } catch(Exception e) {
            string error = string.Format(
                "Error occured while instantiating a ServerAdminDetails instance for queueing. Specificaly while reading bytes from the following file: {0}{1}Exception details follow.{1}{2}",
                pWFileName,
                Initiator.NewLine,
                    e
                );
                Logger.Instance.Log(error);
                throw new Exception(error);
            }
        }

        private static Queue<ServerAdminDetails> QueueServersForShutdown() {

            ServersQueuedForShutdown = new Queue<ServerAdminDetails>();

            const int firstServerIndex = 0;
            int serverCount = firstServerIndex;
            string empty = string.Empty;

            do
            {
                string controller = ConfigReader.Read["Controller" + serverCount];
                string server = ConfigReader.Read["Server" + serverCount];
                string serverPort = ConfigReader.Read["ServerPort" + serverCount];
                string serverUser = ConfigReader.Read["ServerUser" + serverCount];
                string serverUserPwFile = ConfigReader.Read["ServerUserPwFile" + serverCount];

                if (controller == empty || server == empty || serverPort == empty || serverUser == empty || serverUserPwFile == empty)
                    break;

                ServersQueuedForShutdown.Enqueue(
                    new ServerAdminDetails(
                        controller,
                        server,
                        serverPort,
                        serverUser,
                        GetMyCredential(Path.GetFullPath(serverUserPwFile))
                    )
                );

                Logger.Instance.Log (
                    string.Format (
                        "Server admin details of Controller: {0}, Server: {1}, ServerPort: {2}, ServerUser: {3}, ServerUserPwFile: {4} added to queued element number {5}.",
                        controller,
                        server,
                        serverPort,
                        serverUser,
                        serverUserPwFile,
                        serverCount
                    )
                );
                serverCount++;

            } while (true);
            return ServersQueuedForShutdown;
        }

        /// <summary>
        /// Retreives the entropy found in the BinaryMist.PowerOffUPSGuests.dll.config file, used to encrypt all the passwords.
        /// </summary>
        /// <returns>byte[]</returns>
        public static byte[] CredentialEntropy() {
            string[] numbers = ConfigReader.Read["CredentialEntropy"].Split(',');
            byte[] entropy = new byte[numbers.Length];
            for (int i = 0; i < numbers.Length; i++) {
                entropy[i] = Byte.Parse(numbers[i]);
            }
            return entropy;
        }

        /// <summary>
        /// The name of the ServerController child type.
        /// </summary>
        internal string ServerControllerType { get; private set; }

        /// <summary>
        /// The name of the server
        /// </summary>
        internal string ServerName { get; private set; }

        /// <summary>
        /// The port of the server
        /// </summary>
        internal string ServerPort { get; private set; }

        /// <summary>
        /// The user name for the server
        /// </summary>
        internal string UserName { get; private set; }

        /// <summary>
        /// The password for the user
        /// </summary>
        internal byte[] Password { get; private set; }
    }

Back to line 33 of the Initiator.
Now that we can access each ServerAdminDetails within the Queue of ServerAdminDetails via the Queue iterator provided by ServerAdminDetails.QueuedDetails.
We can create the ServerController child types using the late bound Activator.CreateInstance method, based on the ServerAdminDetails.ServerControllerType.
A reference to each ServerController child instance is added to the serverControllers queue.
The Shutdown procedure for each ServerController child is then called.

The ServerController

Notice the constructor which has been called by the child’s constructor, calls back to the child’s AssembleRequests procedure before completing.

    /// <summary>
    /// Controls the process of shutting down the associated server.
    /// </summary>
    /// <remarks>
    /// An instance of this class is created indirectly via the more specific concrete creators for each server that requires shutdown.
    /// Plays the part of the Creator, in the Factory Method pattern.
    /// </remarks>
    internal abstract class ServerController {

        protected static readonly string NewLine = Initiator.NewLine;

        protected enum RequestMethod {
            Get,
            Post
        }

        /// <summary>
        /// Constructor for the <see cref="ServerController"/> class.
        /// Called via the more specific children to initialize the less specific members.
        /// </summary>
        /// <param name="serverAdminDetails">
        /// The details required to create the messages that need to be sent to the server in order to perform the shutdown.</param>
        public ServerController(ServerAdminDetails serverAdminDetails) {
            ServerAdminDetails = serverAdminDetails;
            RequestAssembler = new RequestAssembler();
            SoapEnvelopes = new Queue<XmlDocument>();
            this.AssembleRequests();
        }

        protected ServerAdminDetails ServerAdminDetails { get; set; }

        protected RequestAssembler RequestAssembler { get; set; }

        /// <summary>
        /// Reference a <see cref="System.Collections.Generic.Queue{System.Xml.XmlDocument}">queue</see> of soap envelopes.
        /// used by the children of this class to send to the server.
        /// </summary>
        public Queue<XmlDocument> SoapEnvelopes { get; protected set; }

        /// <summary>
        /// Initial preparation of messages that will be sent to the server to perform shutdown.
        /// </summary>
        /// <remarks>Factory method.</remarks>
        public abstract void AssembleRequests();

        /// <summary>
        /// Completes the compilation of the sequence of messages that need to be sent to the server in order to perform the shutdown.
        /// Sends the messages.
        /// </summary>
        public abstract void Shutdown();

        protected void NotifyOfShutdown() {
            Logger.Instance.Log(
                string.Format(
                    "{0}.Shutdown on server: {1} has now been executed.",
                    ServerAdminDetails.ServerControllerType,
                    ServerAdminDetails.ServerName
                )
            );
        }

        protected bool SimplePing() {
            string serverName = ServerAdminDetails.ServerName;
            Logger.Instance.Log(string.Format("Performing Ping test on server: {0}", serverName));
            Ping pingSender = new Ping();
            int pingRetry = 3;
            PingReply reply = null;
            for (int i = 0; i < pingRetry; i++) {
                try {
                    //may take a couple of tries, as may time out due to arp delay
                    pingSender.Send(serverName);
                    Logger.Instance.Log(string.Format("Initiating Ping number {0} of {1} on server: {2}. ", i + 1, pingRetry, serverName));
                    reply = pingSender.Send(serverName);

                    if (reply.Status == IPStatus.Success) break;

                } catch (Exception e) {
                    Logger.Instance.Log(e.ToString());
                }
            }

            bool optionsAvailable = reply.Options != null;
            string noOptions = "No Options available";

            Logger.Instance.Log(
                "Reply status for server: " + serverName + " was " + reply.Status + ". " + NewLine +
                "Address: " + reply.Address.ToString() + ". " + NewLine +
                "RoundTrip time: " + reply.RoundtripTime + ". " + NewLine +
                "Time to live: " + ((optionsAvailable) ? reply.Options.Ttl.ToString() : noOptions) + ". " + NewLine +
                "Don't fragment: " + ((optionsAvailable) ? reply.Options.DontFragment.ToString() : noOptions) + ". " + NewLine +
                "Buffer size: " + reply.Buffer.Length + ". "
                );

            return reply.Status == IPStatus.Success;
        }
    }

The ServerController children

The AssembleRequests constructs as much of the SOAP envelopes as it can,
without knowing all the information that the target server will provide to be able to complete the messages before being sent.
The RequestAssembler‘s CreateSoapEnvelope does the assembly of the SOAP envelope.
You’ll notice that the RequestAssembler‘s CreateLoginSoapEnvelope takes an extra argument.
The ServerAdminDetails is passed so that the target server’s credentials can be included in the SOAP envelope.
The SOAP envelopes are then queued ready for dispatch.

Now when line 02 or 11 of the Initiator is executed,
line 143 of the VMServerController will be called. That’s Shutdown.
Then we dequeue, send request, and process the response in DequeueSendRequestProcessResponse.
Passing in an optional parameter of Action of HttpWebResponse (the lambda).
From DequeueSendRequestProcessResponse,
we dequeue each SOAP envelope and call CreateWebRequest,
which in-turn, palms the work it knows about to the less specific RequestAssembler‘s CreateWebRequest as a lambda.
Now in DequeueSendRequestProcessResponse, when we get our initialized HttpWebRequest back,
We pass both the SOAP envelope and the HttpWebRequest to the RequestAssembler‘s InsertSoapEnvelopeIntoWebRequest‘s procedure to do the honors.

    /// <summary>
    /// Controls the process of shutting down the associated vSphere server.
    /// </summary>
    /// <remarks>
    /// Plays the part of the Concrete Creator, in the Factory Method pattern.
    /// </remarks>
    internal class VMServerController : ServerController {

        private static string _operationIDEndTag = "</operationID>";
        private static string _operationIDTags = "<operationID>" + _operationIDEndTag;
        private uint _operationIDaVal = 0xAC1CF80C;
        private uint _operationIDbVal = 0x00000000;
        private const int EstimatedHeaderSize = 45;

        private enum RequestType {
            Hello, HandShake, Login, Shutdown
        }

        private string _host;
        private string _uRL;
        private readonly string _action = @"""urn:internalvim25/4.1""";
        private readonly string _userAgent = @"VMware VI Client/4.0.0";
        private KeyValuePair<string, string> _cookie;

        public VMServerController(ServerAdminDetails serverAdminDetails) : base(serverAdminDetails) {

        }

        /// <summary>
        /// Loads the <see cref="ServerController.SoapEnvelopes">SoapEnvelopes</see> with the sequence of messages
        /// that need to be sent to the server in order to perform the shutdown.
        /// </summary>
        /// <remarks>
        /// Factory method implementation
        /// </remarks>
        public override void AssembleRequests() {
            Logger.Instance.LogTrace();
            _host = "https://" + ServerAdminDetails.ServerName + ":" +ServerAdminDetails.ServerPort;
            _uRL = "/sdk";
            SoapEnvelopes.Enqueue(CreateHelloEnvelope());
            SoapEnvelopes.Enqueue(CreateHandshakeEnvelope());
            SoapEnvelopes.Enqueue(CreateLoginEnvelope());
            SoapEnvelopes.Enqueue(CreateShutdownEnvelope());
        }

        private string InitialHeaderContent() {
            StringBuilder headerContent = new StringBuilder(_operationIDTags, EstimatedHeaderSize);
            headerContent.Insert(headerContent.ToString().IndexOf(_operationIDEndTag), _operationIDaVal.ToString("X8") + "-" + (++_operationIDbVal).ToString("X8"));
            return headerContent.ToString();
        }

        private XmlDocument CreateHelloEnvelope() {
            Logger.Instance.LogTrace();
            string bodyContent = @"
    <RetrieveServiceContent xmlns=""urn:internalvim25"">
      <_this xsi:type=""ManagedObjectReference"" type=""ServiceInstance"" serverGuid="""">ServiceInstance</_this>
    </RetrieveServiceContent>";
            return RequestAssembler.CreateSoapEnvelope(InitialHeaderContent(), bodyContent);
        }

        private XmlDocument CreateHandshakeEnvelope() {
            Logger.Instance.LogTrace();
            string bodyContent = @"
    <RetrieveInternalContent xmlns=""urn:internalvim25"">
      <_this xsi:type=""ManagedObjectReference"" type=""ServiceInstance"" serverGuid="""">ServiceInstance</_this>
    </RetrieveInternalContent>";
            return RequestAssembler.CreateSoapEnvelope(InitialHeaderContent(), bodyContent);
        }

        private XmlDocument CreateLoginEnvelope() {
            Logger.Instance.LogTrace();
            string bodyContent = @"
    <Login xmlns=""urn:internalvim25"">
      <_this xsi:type=""ManagedObjectReference"" type=""SessionManager"" serverGuid="""">ha-sessionmgr</_this>
      <userName></userName>
      <password></password>
      <locale>en_US</locale>
    </Login>";
            try {
                // As VMware insist on putting credentials in the SOAP body what else can we do?
                return RequestAssembler.CreateLoginSoapEnvelope(InitialHeaderContent(), bodyContent, ServerAdminDetails);
            } catch(InvalidCredentialException e) {
                string error = string.Format(
                    "Error occured during a call to CreateLoginSoapEnvelope, for server: {0}.{1}Exception details follow.{1}{2}",
                    ServerAdminDetails.ServerName,
                    NewLine,
                    e
                );
                Logger.Instance.Log(error);
                throw new Exception(error);
            }
        }

        private XmlDocument CreateShutdownEnvelope() {
            Logger.Instance.LogTrace();
            string bodyContent = @"
    <ShutdownHost_Task xmlns=""urn:internalvim25"">
      <_this xsi:type=""ManagedObjectReference"" type=""HostSystem"" serverGuid="""">ha-host</_this>
      <force>true</force>
    </ShutdownHost_Task>";
            return RequestAssembler.CreateSoapEnvelope(InitialHeaderContent(), bodyContent);
        }

        private HttpWebRequest CreateWebRequest(string uRL, KeyValuePair<string, string> cookie) {
            return RequestAssembler.CreateWebRequest(
                uRL,
                (request)=>{
                    request.Method = RequestMethod.Post.ToString();
                    request.UserAgent = _userAgent;
                    request.ContentType = "text/xml; charset=\"utf-8\"";
                    request.Headers.Add("SOAPAction", _action);
                    request.Accept = "text/xml";
                    request.KeepAlive = true;

                    if (!string.IsNullOrEmpty(cookie.Key))
                        request.Headers.Add("Cookie", cookie.Key + "=" + cookie.Value);
                }
            );
        }

        private void DequeueSendRequestProcessResponse(RequestType requestType, KeyValuePair<string, string> cookie, Action<HttpWebResponse> additionalResponseProcessing = null) {
            Logger.Instance.Log(string.Format("Will now attempt sending {0} message to server: {1}", requestType, ServerAdminDetails.ServerName));
            XmlDocument soapEnvelope = SoapEnvelopes.Dequeue();
            HttpWebRequest httpWebRequest = CreateWebRequest(_host + _uRL, cookie);
            RequestAssembler.InsertSoapEnvelopeIntoWebRequest(soapEnvelope, httpWebRequest);

            string soapResult;
            using (HttpWebResponse response = (HttpWebResponse)httpWebRequest.GetResponse())
            using (Stream responseStream = response.GetResponseStream())
            using (StreamReader streamReader = new StreamReader(responseStream)) {
                //pull out the bits we need for the next request.

                if (additionalResponseProcessing != null) {
                    additionalResponseProcessing(response);
                }

                soapResult = streamReader.ReadToEnd();
            }
        }

        /// <summary>
        /// Perform the shutdown of the server specified in the <see cref="ServerAdminDetails">server admin details</see>.
        /// </summary>
        public override void Shutdown() {

            bool serverOnline = SimplePing();
            Logger.Instance.Log(
                serverOnline
                    ? string.Format("Initiating sending of {0} to server: {1}", RequestType.Hello, ServerAdminDetails.ServerName)
                    : string.Format("Could not reach server: {0}. Aborting shutdown of server: {0}", ServerAdminDetails.ServerName)
            );

            ServicePointManager.ServerCertificateValidationCallback += ValidateRemoteCertificate;

            KeyValuePair<string, string> emptyCookie = new KeyValuePair<string, string>();
            DequeueSendRequestProcessResponse(
                RequestType.Hello,
                emptyCookie,
                (response)=> {
                    string[] setCookieElementsResponse = response.Headers["Set-Cookie"].Split(new[] { "\"" }, StringSplitOptions.RemoveEmptyEntries);
                    _cookie = new KeyValuePair<string, string>(setCookieElementsResponse[0].TrimEnd('='), "\"" + setCookieElementsResponse[1] + "\"");
                }
            );
            DequeueSendRequestProcessResponse(RequestType.HandShake, _cookie);
            DequeueSendRequestProcessResponse(RequestType.Login, _cookie);
            DequeueSendRequestProcessResponse(RequestType.Shutdown, _cookie);
            NotifyOfShutdown();
        }

        private static string RemoteCertificateDetails(X509Certificate certificate) {
            return string.Format(
                "Details of the certificate provided by the remote party are as follows:" + NewLine +
                "Subject: {0}" + NewLine +
                "Issuer: {1}",
                certificate.Subject,
                certificate.Issuer
            );
        }
    }

The RequestAssembler

    /// <summary>
    /// Provides ancillary operations for <see cref="ServerController">server controllers</see>
    /// that assist in the creation of the requests intended for dispatch to the servers requiring shutdown.
    /// </summary>
    internal class RequestAssembler {

        private static string _soapEnvelope = @"<soap:Envelope xmlns:xsd='http://www.w3.org/2001/XMLSchema' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns:soap='http://schemas.xmlsoap.org/soap/envelope/'>
    <soap:Header>
    </soap:Header>
    <soap:Body>
    </soap:Body>
</soap:Envelope>";

        /// <summary>
        /// Produces a SOAP envelope in XML form with the server admins user credentials
        /// </summary>
        /// <param name="headerContent">Content targeted for between the header tags.</param>
        /// <param name="bodyContent">Content targeted for between the body tags.</param>
        /// <param name="serverAdminDetails">The <see cref="ServerAdminDetails"/> instance used to access the user name and password.</param>
        /// <returns>The SOAP envelope.</returns>
        public XmlDocument CreateLoginSoapEnvelope(string headerContent, string bodyContent, ServerAdminDetails serverAdminDetails) {

            string failMessage = "The credentials were not correctly set. Username: {0} Password byte count: {1}";
            if (string.IsNullOrEmpty(serverAdminDetails.UserName) || serverAdminDetails.Password.Length < 1)
                throw new InvalidCredentialException(string.Format(failMessage, serverAdminDetails.UserName, serverAdminDetails.Password.Length));

            StringBuilder bodyContentBuilder= new StringBuilder(bodyContent, bodyContent.Length + 10);
            bodyContentBuilder.Insert(
                bodyContentBuilder.ToString().IndexOf("</userName>"),
                serverAdminDetails.UserName
            );

            bodyContentBuilder.Insert(
                bodyContentBuilder.ToString().IndexOf("</password>"),
                decryptedCredential(serverAdminDetails)
            );

            return CreateSoapEnvelope(headerContent, bodyContentBuilder.ToString());
        }

        private string decryptedCredential(ServerAdminDetails serverAdminDetails) {
            try {
                return new ASCIIEncoding().GetString(
                    ProtectedData.Unprotect(
                        serverAdminDetails.Password,
                        ServerAdminDetails.CredentialEntropy(),
                        DataProtectionScope.CurrentUser
                    )
                );
            } catch(CryptographicException e) {

                // Retrieve the exception that caused the current
                // CryptographicException exception.
                Exception innerException = e.InnerException;
                string innerExceptionMessage = "";
                if (innerException != null) {
                    innerExceptionMessage = innerException.ToString();
                }

                // Retrieve the message that describes the exception.
                string message = e.Message;

                // Retrieve the name of the application that caused the exception.
                string exceptionSource = e.Source;

                // Retrieve the call stack at the time the exception occured.
                string stackTrace = e.StackTrace;

                // Retrieve the method that threw the exception.
                System.Reflection.MethodBase targetSite = e.TargetSite;
                string siteName = targetSite.Name;

                // Retrieve the entire exception as a single string.
                string entireException = e.ToString();

                // Get the root exception that caused the current
                // CryptographicException exception.
                Exception baseException = e.GetBaseException();
                string baseExceptionMessage = "";
                if (baseException != null) {
                    baseExceptionMessage = baseException.Message;
                }

                Logger.Instance.Log(
                    "Caught an unexpected exception:" + Initiator.NewLine
                    + entireException + Initiator.NewLine
                    + Initiator.NewLine
                    + "Properties of the exception are as follows:" + Initiator.NewLine
                    + "Message: " + message + Initiator.NewLine
                    + "Source: " + exceptionSource + Initiator.NewLine
                    + "Stack trace: " + stackTrace + Initiator.NewLine
                    + "Target site's name: " + siteName + Initiator.NewLine
                    + "Base exception message: " + baseExceptionMessage + Initiator.NewLine
                    + "Inner exception message: " + innerExceptionMessage + Initiator.NewLine
                );
                throw;
            }
        }

        /// <summary>
        /// Produces a SOAP envelope in XML form.
        /// </summary>
        /// <param name="headerContent">Content targeted for between the header tags.</param>
        /// <param name="bodyContent">Content targeted for between the body tags.</param>
        /// <returns><see cref="System.Xml.XmlDocument">The SOAP envelope</see>.</returns>
        public XmlDocument CreateSoapEnvelope(string headerContent, string bodyContent) {

            StringBuilder sb = new StringBuilder(_soapEnvelope);

            try {
                sb.Insert(sb.ToString().IndexOf(Initiator.NewLine + "    " + "</soap:Header>"), headerContent);
                sb.Insert(sb.ToString().IndexOf(Initiator.NewLine + "    " + "</soap:Body>"), bodyContent);
            } catch(Exception e) {
                Logger.Instance.Log(e.ToString());
                throw;
            }

            XmlDocument soapEnvelopeXml = new XmlDocument();
            soapEnvelopeXml.LoadXml(sb.ToString());

            return soapEnvelopeXml;
        }

        /// <summary>
        /// Creates a web request based on the url passed in.
        /// </summary>
        /// <param name="url">The target URL of the server that will be shutdown.</param>
        /// <param name="additionalWebRequestManipulation">delegate of type
        /// <see cref="System.Action{HttpWebRequest}">Action{HttpWebRequest}</see>.
        /// This is used to take additional tasking defined in the calling procedure.
        /// This procedure creates the web request,
        /// and passes it into this parameter for the additional initialization work to be performed on the web request.
        /// </param>
        /// <returns>
        /// An initialized <see cref="System.Net.HttpWebRequest">web request</see>,
        /// ready to have a <see cref="System.Xml.XmlDocument">soap envelope</see> inserted.</returns>
        public HttpWebRequest CreateWebRequest(string url, Action<HttpWebRequest> additionalWebRequestManipulation = null) {
            HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(url);

            if(additionalWebRequestManipulation != null) {
                additionalWebRequestManipulation(webRequest);
            }
            return webRequest;
        }

        /// <summary>
        /// Insert the <see cref="System.Xml.XmlDocument">soap envelope</see> into the <see cref="System.Net.HttpWebRequest">web request</see>.
        /// </summary>
        /// <param name="soapEnvelopeXml">
        /// The <see cref="System.Xml.XmlDocument">soap envelope</see> to be inserted into the
        /// <see cref="System.Net.HttpWebRequest">web request</see>.
        /// </param>
        /// <param name="webRequest">The <see cref="System.Net.HttpWebRequest">web request</see> that the
        /// <see cref="System.Xml.XmlDocument"/>soap envelope</see> is inserted into.
        /// </param>
        public void InsertSoapEnvelopeIntoWebRequest(XmlDocument soapEnvelopeXml, HttpWebRequest webRequest) {
            using (Stream stream = webRequest.GetRequestStream()) {
                soapEnvelopeXml.Save(stream);
            }
        }

        private static void InsertByteArrayIntoWebRequest(byte[] postData, HttpWebRequest webRequest) {
            using (Stream stream = webRequest.GetRequestStream()) {
                stream.Write(postData, 0, postData.Length);
            }
            webRequest.ContentLength = postData.Length;
        }

        /// <summary>
        /// Inserts the credentials from the <see cref="ServerAdminDetails">server admmin details</see> into the
        /// <see cref="System.Net.HttpWebRequest">web request</see> supplied.
        /// </summary>
        /// <param name="serverAdminDetails">
        /// The <see cref="ServerAdminDetails">server admin details</see>
        /// containing the information for the server that the request will be sent to.
        /// </param>
        /// <param name="webRequest">
        /// The <see cref="System.Net.HttpWebRequest">web request</see> that will have the server administration credentials inserted.
        /// </param>
        public void InsertCredentialsIntoWebRequest(ServerAdminDetails serverAdminDetails, HttpWebRequest webRequest) {
            InsertByteArrayIntoWebRequest(
                new ASCIIEncoding().GetBytes("username=" + serverAdminDetails.UserName + "&password=" + decryptedCredential(serverAdminDetails)),
                webRequest
             );
        }
    }


Let me know if any of this is unclear, and requires additional explanation.

Once again, the full source code can be found here.

Preparing APC Smart-UPS 1500 clients

July 26, 2011

Part two of a three part series

on Setting up a UPS solution, to enable clean shutdown of vital network components.

This post is about setting up the software that will be responsible for cleanly shutting down servers and workstations.

We have to decide which machine/s is/are going to be used to launch our script (which in turn is run by what APC call a command file).

Currently I’ve got an old laptop I pulled out of the rubbish about 5 years ago, with Windows XP running on it.
It’s got just enough battery capacity to stay alive for long enough to receive the event from the NMC (Network Management Card) and run my .dll that issues the shutdown sequence.
A couple of EeePC 901’s have also recently been made redundant, and I may use one of those with Windows 7 installed at some stage.
Currently all of my workstations and servers that don’t have batteries, I.E. notebooks are VM’s running on ESXi.
Oh… or servers that have their entire file system loaded into volatile memory, so if they are powered off, I.E. cold shutdown, there is no possible corruption of the file system.
What you can also do is host the PCNS (PowerChute Network Shutdown) on a VM, because once the shutdown of ESXi has been initiated, there is no stopping the sequence, and the VM’s will all be cleanly shutdown.
Or better still, use more than one machine to host PCNS, as they will operate on a first in first served basis.
As you’ll see here the NMC’s firmware and PCNS are quite extensible.
The above document is recommended reading if your planning on setting up an APC UPS and want to automate clean shutdowns.
Without reading, the comms can get a little confusing.

Setting up PCNS

Install the PowerChute Network Shutdown service

You can get a copy of v2.2.3 here
I later found out that there were later versions:
v2.2.4 linked to from here, which has additional documentation.
v3.0.0 linked to from here, which has additional documentation.
Both of which were linked to from here, which has additional manuals etc.
You can find the installation guide here.
The PCNS service needs to be run as a local Administrator as the default Local System account doesn’t have sufficient rights.
In saying all that, William Tournas from APC recommended I use PCNS 2.2.1 for Windows XP.
Additional 2.2.1 resources are found here.

If using vMA with PCNS 3.0, you go through a Web UI configuration wizard once installed.
If using PCNS with Windows, the configuration is part of the install.

Either way, the steps will look similar to the following:

netstat -a

Should show that PCNS is listening on TCP and UDP ports 3052

If it’s not, you’ll need to open those ports on your firewall.

If you’re looking at using a Linux based VM to host PCNS,
VMware provide vMA (vSphere Management Assistant) a CentOS VM image.
You can get the binary here.
You’ll also need PCNS.
Take your pick of the following binaries:
2.2.4
3.0.0
Along with the documentation:
PowerChute_NetworkShutdownv2.4-ReleaseNotes.htm
PowerChute_NetworkShutdownv3.0-ReleaseNotes.htm
You’ll have to have the same ports open, as PCNS will be listening on them.
A listing of iptables for the filter (default unless otherwise specified) table should look like the following:

For an easier to read output, try the following:

sudo iptables  -L -v -n --line-numbers | column -t

Once again, if these ports are not open, you’ll have to find which script is being used to set up the rules.
I’m not sure about CentOS, but in a Debian based system, you would normally put the firewall init script in /etc/init.d/
This script would call a script that sets up the rules and one that tears them down.
I’m going to be making a post about how I set up my firewall (iptables) rules for the netfilter module on our notebooks at some stage soon.
If I haven’t already done this and you need more help, just sing out.

I found the following links quite helpful with the setup:

Link1

ESXi.pdf linked to from here linked to from here.

This has a list of the ports that are supposed to be open on pcns 2.2.3 with ESX 3.5
I think this also applies to PCNS 3.0 and ESXi 4.1 which I tried out.

Also be aware that there’s a known issue with special characters in the credentials for PCNS 3.0

I read somewhere that the PCNS needs to have the same credentials as the NMC, so just be aware of this.

Could be useful for trouble shooting vMA (vSphere Management Assistant)
I made a couple of posts there.

PowerChute Network Shutdown v3.0 – Release Notes
goes through a whole lot of issues and work-arounds with PowerChute.
For example, discuss’s the correct way to run the command file, PowerOff.bat in our case.

The APC PCNS receives an event from the AP9606 (that’s the NMC (Network Management Card)) fitted to the UPS.
The script is launched by APC PCNS from a Windows or Linux box.
I read that PCNS will always shutdown the windows machine it’s running on.
This is not true.

My attempt at using a PowerShell script utilizing mostly VMware’s cmdlets to shutdown ESXi

PowerChute has an option to ‘run this command’ but it’s limited to 8.3 paths and won’t accept command line parameters.
A separate batch file is needed (I called it poweroff.bat)
that runs the shutdown script with the parameters – but that could shut down other ESXi boxes as well if required.

I was keen to use PowerShell to perform the shutdowns, as I’d read it was quite capable and also VMware supplied a large set of management cmdlets.

Install PowerCLI from here.
read the installation guide.
As an admin, run the following:

set-executionpolicy remotesigned

Details of executionpolicy here.

If running PowerOffUPSGuests.ps1 from command shell rather than from a batch file.
You need to add the PowerCLI snapin.

PS C:\scripts&gt; Add-PSSnapin VMware.VimAutomation.Core
PS C:\scripts&gt; . .\PowerOffUPSGuests.ps1 MyESXiHostName AdminUserName

This will establish the SSL connection to MyESXiHostName

Following are the PowerShell scripts I used.

First we had to create our password file to use to log in to vSphere.
See this post for how this was done.

PowerOffUPSGuests.bat (the command file)

echo off
REM VMware would have used the Export-Console cmdlet to export the name of the PowerShell snap-in PowerCLI uses.
REM to the PowerShell console file (.psc1)

REM Invoke the command with the call operator (The ampersand).
PowerShell.exe -PSConsoleFile "C:\Program Files\VMware\Infrastructure\vSphere PowerCLI\vim.psc1" "& "C:\Scripts\PowerOffUPSGuests.ps1" MyESXiServer.MyDomain MyUser

PowerOffUPSGuests.ps1 (the script that was going to do the work)

param ( [parameter(Mandatory=$true)][string] $vSphereServername,
   [parameter(Mandatory=$true)][string] $user
)

$HostCredential = C:\Scripts\Get-myCredential.ps1 $user C:\Scripts\mp.txt

Set-StrictMode -Version 2.0
Write-Host "Establishing connection to $vSphereServername" -ForegroundColor Yellow
Connect-VIServer -Server $vSphereServername -Protocol https -Credential $HostCredential

function Stop-VMOnVMHost {
   Write-Host "Shutting down guests." -ForegroundColor Yellow

   $vM = Get-VM | Where-Object {$_.PowerState -eq "PoweredOn" -and $_.Guest.State -eq "Running"}
   Write-Host "Shutting down the following guests: $vM " -ForegroundColor Yellow
   $vM | Shutdown-VMGuest -Confirm:$False
   $seconds = 300
   Write-Host "Waiting $seconds Seconds. "
   Start-Sleep -Seconds $seconds

   $vM = Get-VM | Where-Object {$_.PowerState -eq "PoweredOn"}
   Write-Host "Stopping the following guests: $vM " -ForegroundColor Yellow
   $vM | Stop-VM -RunAsync -Confirm:$False
   $seconds = 60
   Write-Host "Waiting %seconds Seconds. "
   Start-Sleep -Seconds $seconds
}

function Stop-VMHost {
   Write-Host "Setting state of $vSphereServername to maintenance mode. " -ForegroundColor Yellow
   Get-VMHost | ForEach-Object {
      $hostName = $_.Name
      Write-Host "Putting $hostName into maintenance mode. "
      Set-VMHost -vmhost $_ -state maintenance
      Write-Host "Stopping $hostName. "
      Stop-VMHost -vmhost $_ -RunAsync
   }
}

Stop-VMOnVMHost
Stop-VMHost
Write-Host "Shutdown Complete" -ForegroundColor Yellow

Tried my script and got the following:

Shutdown-VMGuest     Operation “Shutdown VM guest.” failed for VM “MyGuestNameHere” for the following reason: fault.Restriction.summary

I had a hunch that it was due to the read only restriction I had heard about.
So tried command straight from PowerShell console…
same result.
More details here.
PowerCLI references to shutting down ESX
http://pastebin.com/HgsbSpb7
http://www.sheenaustin.com/2011/02/20/vmware-ups-shutdown-script/
http://communities.vmware.com/message/1555286
http://spininfo.homelinux.com/news/vSphere_PowerCLI/2010/06/18/Shutdown_infrastructure_on_power_outage
http://blogs.vmware.com/kb/2010/09/managing-esxi-made-easy-using-powercli.html
http://www.vmware.com/support/developer/PowerCLI/PowerCLI41U1/html/index.html

So as it turned out, VMware has removed write access from PowerCLI to ESXi, in 4.0 onwards I think.

Back to Scripting SOAP

As I was kind of out of luck with using PowerCLI cmdlets,
I decided to write my own library,
that I would execute using PowerShell.

First command needs to shutdown my fileserver.
Issue:
hello, authenticate, shutdown

Used Burp suite to diagnose the http frames being sent received from/to vSphere client/ESXi.
I haven’t used this tool before, but it gave very good visibility of the messages being sent/received.
The vSphere client has a config file here:
C:\Program Files\VMware\Infrastructure\Virtual Infrastructure Client\Launcher\VpxClient.exe.config
that you can change the ports that the vSphere client sends/receives on,
but I found it easier to just set the IP address / Name field in the GUI
to point to 127.0.0.1:8080 This is where the Burp proxy listens on by default.

In Burp, you will also need to add another proxy listener to the proxy->options tab.
Set Local Listener to 8080,
Uncheck listen on loopback interface only,
Check support invisible proxying for non-proxy-aware clients.
The in-app help has good documentation on this.
Set redirect to host to the ESXi host name.
Set redirect to port to the ESXi’s default SSL port of 443
Select the generate CA-signed per-host certificates radio button.

I also made sure the new proxy rule was the only one running.
When Burp captures each frame, you can forward each one onto any one of the other tools in the suite.
This is a really nice tool.

My PowerOffUPSGuests.ps1 was about to significantly change too.
Running my PowerOffUPSGuests.ps1 script using PowerShell

PS C:\Scripts\UPS&gt; . ".\PowerOffUPSGuests.ps1"

We no longer need to pass any arguments to PowerOffUPSGuests.ps1

I was going to be using .net 4 libraries in my PowerOffUPSGuests.dll,
so needed to Let PowerShell know about the .net 4 CLR.
By default PS 2.0 is only aware of the .net 2.0 framework.

Some insight on this:
http://stackoverflow.com/questions/2094694/launch-PowerShell-under-net-4
http://tfl09.blogspot.com/2010/08/using-newer-versions-of-net-with.html
http://www.powergui.org/thread.jspa?threadID=13403

So needed to create a couple of config files for
%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\PowerShell.exe
and
PowerShell_ise.exe
with config appended
%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\PowerShell.exe.config
and
PowerShell_ise.exe.config
with the following contents:

<?xml version="1.0"?>
<configuration>
  <startup useLegacyV2RuntimeActivationPolicy="true">
     <supportedRuntime version="v4.0.30319"/>
     <supportedRuntime version="v2.0.50727"/>
   </startup>
</configuration>


This works for PowerShell, but not for PowerGUI (obviously) which I was using for debugging.


So If you still need PowerGUI you’ll have to add the registry hacks explained in the links above.
Remember to remove them once finished as they take affect system wide.

I also had some trouble with later versions of C# than 2.0 when compiling on the fly in PowerShell.
Although I was specifying the language.

Add-Type -Path $typePath -CompilerParameters $compilerParameters -Language csharpversion3


Found a workaround this bug here.

# add the block of code we call into
$code = [io.file]::ReadAllText((Join-Path -Path $scriptPath -ChildPath $powerOffUPSGuestsFile))
Add-Type $code -CompilerParameters $compilerParameters -Language CSharpVersion3


We’ll go over the library code in the third part of this series.

As it stands now, the C:\Scripts\UPS\PowerOff.bat looks like this

echo.
echo PowerOffUPSGuests.ps1 started at the following time: %time% &gt;&gt; C:\Scripts\UPS\Log.txt
"C:\WINDOWS\system32\WindowsPowerShell\v1.0\PowerShell.exe" C:\Scripts\UPS\PowerOffUPSGuests.ps1
echo PowerOffUPSGuests.ps1 finished at the following time: %time% &gt;&gt; C:\Scripts\UPS\Log.txt
echo.


The PowerOffUPSGuests.ps1 looks like this

Set-StrictMode -Version 2.0

# add the assembly that does the work.
Add-Type -Path C:\Scripts\UPS\PowerOffUPSGuests.dll

# instantiate a PowerOffUPSGuests instance
$powerOffUPSGuestsInstance = New-Object -TypeName BinaryMist.Networking.Infrastructure.PowerOffUPSGuests

Write-Host $powerOffUPSGuestsInstance.InitShutdownOfServers() -ForegroundColor Green

The essential files

Testing that everything works

I was unsure whether we were going to be able to get ESXi to cleanly shutdown it’s guest VM’s.
As I’d had some trouble with this previously.

I was thinking about writing a WCF service and client to shutdown windows guests for now.
The service being on the machine that needed to be cleanly shutdown.
Could use something like the following command line in the service.

shutdown.exe -m //MachineNeedingShutdown -t 10 -c "Shutting down due to UPS running on battery." -s -f

Wrapped in something like this…

Process shutdownMyBox = new Process();
shutdownMyBox.StartInfo.FileName = shutdown.exe;
shutdownMyBox.StartInfo.Arguments = "-m //MachineNeedingShutdown -t 10 -c \"Shutting down due to UPS running on battery.\" -s -f";
shutdownMyBox.Start();

I was sure there was a better way though.

The sequence of events I was thinking of was something like the following:

First we try to shutdown every VM guest, set vMGuestTimmer
If all VM guests shutdown
——try put host into maintenance mode, set timer.
——when in maintenance mode
———shutdown host
——if enter maintenance mode not successful within time set
———shutdown host
On vMGuestTimmer
——shutdown host

There was a better/easier way though

In the PCNS Web UI -> PowerChute->MachineName->Configure Events
You can set PowerOff.bat to run after 30 seconds, or for testing,
set it to something really small, so it runs the command file sooner.
Set the time that’s required for the command file to complete to 5 minutes.
Although I don’t think it matters that much, as long as there’s enough time to start the execution of the PowerShell script.
Once the script is running, we don’t care how long PCNS thinks it should wait, as it’s non blocking.

To test that pcns will run your batch file:
Just put some temporary script, something like the following in your
C:\Scripts\UPS\PowerOff.bat

time/T &gt;&gt; C:\Scripts\UPS\MyTest.txt

These are some links from APC to help get your PCNS command file running:
http://jpaa-en.apc.com/app/answers/detail/a_id/7712
http://jpaa-en.apc.com/app/answers/detail/a_id/2441
http://jpaa-en.apc.com/app/answers/detail/a_id/1175


What is needed for ESXi to shut down all machines cleanly?

Graceful shutdown work around for ESXi guests.

Also it’s important to make sure the root user of ESXi has the Administrator Role.

What is needed for Windows VM’s to shutdown cleanly?

First ascertain whether or not your VM is/isn’t being shutdown cleanly.
eventvwr is your friend.

The scripts that may play a part in the shutting down of the Windows VM’s.
If you have a look at the VMware Tools Properties->Scripts tab
You can see for the shutdown script, that it actually does nothing.
If you find that your Windows box is not shutting down cleanly…
Add a custom script to the “Shut Down Guest Operating System” Script Event
I just created a shutdown.bat with the following in it.

C:\Windows\System32\shutdown.exe -s -t 1

This cleared up any errors I was getting in my Windows7 logs.

What is needed for Linux VM’s to shutdown cleanly?

If you’re looking at Debian based systems…
View the relevant log that contains shutdown info.

sudo vi /var/log/messages

and

sudo vi /var/log/syslog

From command mode (that’s [Esc]) to show line numbers,

type

:set number

or

:set nu

To find the matches for “shutdown” (without quotes) ignoring case

sudo grep -i -n "shutdown" /var/log/messages

Or easier still…
Once the file’s open in vi,
From command mode

/shutdown

[n]            will repeat the search forward
[N]            will repeat the search in opposite direction

My Debian wheezy server wasn’t getting shutdown cleanly.
So tried to install vmware tools, but found the easier way was to use open-vm-tools
Added contrib to my /etc/apt/sources.list
Installed open-vm-tools open-vm-source
Had some trouble with the NZ repo for those packages, they were corrupt.
So renamed /etc/apt/apt.conf so apt-get wasn’t using my cached packages from apt-cacher.

sudo apt-get clean
sudo apt-get update
sudo apt-get install open-vm-tools open-vm-source

The scripts that may play a part in the shutting down of the Linux VM’s.
Read this link.
There are also vmware-tools scripts
Mine didn’t appear to do much, but my server was being shutdown cleanly now.


Shout out if anythings unclear.

In part three I’ll be going over the library I’ve written that actually does the work 😉

Using PSCredentials

June 2, 2011

I’ve been working on a small project that shuts down machines attached by network and of course power feed to an APC Smart-UPS.
The code that was shutting down the guests required authentication to be passed to the receiving services.

I decided to give the following PowerShell cmdlets a try.

  • Get-Credential
  • ConvertTo-SecureString

———————————————————————————-

Script that creates the password file

(Set-Credential.ps1) looks like this:

Param($file)
$credential = Get-Credential
$credential.Password | ConvertFrom-SecureString | Set-Content $file

Get-Credential prompts for a username and password and creates the PSCredential associating the password with the username.
ConvertFrom-SecureString from the PS documentation…
The ConvertFrom-SecureString cmdlet converts a secure string
(System.Security.SecureString) into an encrypted standard string (System.String).
Then writes the string to the file specified.

Set-Credential can be invoked like this:

C:\Scripts\UPS\Set-myCredential.ps1 C:\Scripts\UPS\mp.txt

———————————————————————————-

Script that reads the password file

(Get-Credential.ps1) into a SecureString.
Then creates the PSCredential based on the username provided and the password as a SecureString.
Then returns the PSCredential:

Param($user,$passwordFile)
$password = Get-Content $passwordFile | ConvertTo-SecureString
$credential = New-Object System.Management.Automation.PsCredential($user,$password)
$credential

———————————————————————————-

By the look of it, when creating the encrypted password Get-Credential adds some machine specific information.
As the password file is not machine agnostic (can’t be shared or tranfered).

From my PowerShell script that loaded the assembly into memory and started the shutdown procedure, it looked something like this.

param (
   [parameter(Mandatory=$true, position=0)][string] $scriptPath,
   [parameter(Mandatory=$true, position=1)][string] $fileServerName,
   [parameter(Mandatory=$true, position=2)][string] $fileServerUser,
   [parameter(Mandatory=$true, position=3)][string] $vSphereServerName,
   [parameter(Mandatory=$true, position=4)][string] $vSphereServerUser
)

Set-StrictMode -Version 2.0
# Creates a .net assembly in memory containing the PowerOffUPSGuests class.
# Then we call the InitShutdown passing the details of the machines that need to be shutdown.

$credentialRetrievalScript = Join-Path -Path $scriptPath -ChildPath 'Get-Credential.ps1'
$fileServerUserPwLocation = Join-Path -Path $scriptPath -ChildPath 'FileServerPw.txt'
$vSphereServerUserPwLocation = Join-Path -Path $scriptPath -ChildPath 'VMHostPw.txt'

# names of the ServerController's I.E. the collection of servers that will be shutdown
# these class's need to exist in the $scriptPath and derive from ServerController
$freeNASController = 'FreeNASController'
$vMServerController = 'VMServerController'

# instantiate the credential objects
$fileServerCredential = & $credentialRetrievalScript $fileServerUser $fileServerUserPwLocation
$vSphereServerCredential = & $credentialRetrievalScript $vSphereServerUser $vSphereServerUserPwLocation

# add the assembly that does the work.
Add-Type -Path .\PowerOffUPSGuests.dll

# instantiate a ServerAdminDetails for each server we want to shutdown
$fileServerAdminDetailsInstance = New-Object -TypeName BinaryMist.Networking.Infrastructure.ServerAdminDetails -ArgumentList $freeNASController, $fileServerName, $fileServerCredential
$vSphereServerAdminDetailsInstance = New-Object -TypeName BinaryMist.Networking.Infrastructure.ServerAdminDetails -ArgumentList $vMServerController, $vSphereServerName, $vSphereServerCredential

# instantiate a PowerOffUPSGuests
$powerOffUPSGuestsInstance = New-Object -TypeName BinaryMist.Networking.Infrastructure.PowerOffUPSGuests

# create generic queue and populate with each of the ServerAdminDetail items
# ServerAdminDetails is the base class of FileServerAdminDetails and vSphereServerAdminDetails
$serverAdminDetailsQueueInstance = .\New-GenericObject System.Collections.Generic.Queue BinaryMist.Networking.Infrastructure.ServerAdminDetails
$serverAdminDetailsQueueInstance.Enqueue($fileServerAdminDetailsInstance)
$serverAdminDetailsQueueInstance.Enqueue($vSphereServerAdminDetailsInstance)

$powerOffUPSGuestsInstance.InitShutdownOfServers($serverAdminDetailsQueueInstance)

To debug my library code, I needed to run it somehow.
So I just wrote a small test which passed the PSCredential instance to the code that was going to shutdown the UPS guest.

private PSCredential GetMyCredential(string userName, string pWFileName) {

   string encryptedPw;
   using (StreamReader sR = new StreamReader(pWFileName)) {
   //read the encrypted bytes into a string
   encryptedPw = sR.ReadLine();
   }

   PSCredential pSCredential;
   using(SecureString pW = new SecureString()) {
      char[] pWChars = encryptedPw.ToCharArray();
      foreach(char pWChar in pWChars) {
         pW.AppendChar(pWChar);
      }
      pSCredential = new PSCredential(userName, pW);
   }
   return pSCredential;
}

[TestMethod]
public void TestInitFileServerShutdown() {
   _powerOffUPSGuests = new PowerOffUPSGuests(ConfigurationManager.AppSettings[LogFilePath]);

   PSCredential fileServerCredential = GetMyCredential(
      ConfigurationManager.AppSettings[FileServerUser],
      Path.GetFullPath(ConfigurationManager.AppSettings[FileServerUserPwFile])
   );

   _powerOffUPSGuests.InitFileServerShutdown(ConfigurationManager.AppSettings[FileServer], fileServerCredential);
}

Inspiration

VisualHG works nicely in VS 2010 rtm

November 20, 2010

 

For the uninitiated…
VisualHG is a Mercurial source control plugin for MS Visual Studio

 

 

 

 

I had a bit of a problem with VisualHG a couple of weeks ago.
The overlay icons work in Visual Studio, but the context menu had no VisualHG icons and the VisualHG toolbar was inactive (had all the buttons greyed out).

Tried uninstalling / reinstalling visualhg.
Version: Visual Studio 2010 Version 10.0.30319.1 RTMRel Premium.
OS used: Windows 7 Enterprise x64.
VisualHG 1.1.0 installed.
Seems to work in VS 2008 though.

Solution

Remove all instances of VisualHG in Programs and Features.
Search the Program Files and Program Files (x86) for instances of visualhg and remove if they exist.
Search the registry for VisualHG related items (entries as well as keys). Delete them all.
Reinstall the latest version of VisualHG (currently 1.1.0).

Posts and links that provided resolution:
http://visualhg.codeplex.com/Thread/View.aspx?ThreadId=209792
http://visualhg.codeplex.com/workitem/36

Java C# Similarities

November 14, 2010
Part 1

In this post, I’m just going to go over setting up command line compilers for C# and Java.

I was recently helping someone with a Java based project, so had to brush up on my skills in that area.
I haven’t had a need to use Java for a while.
Java is quite similar to C# in many ways.
When it comes to working out the similarities, I think it’s actually easier to work out the differences, because there are less of them.
It often feels like C# is just a more mature Java, with more features and bigger libraries.
In saying that, Java also has benefits over C#.
We’ll explore some of these in this series.

Without further ado, let’s get started.
The first thing I looked at doing, which I’d been meaning to for a while, was to set up command line compilation for both C# and Java on Windows 7.
The idea is that you can just run a command prompt and the directory that cmd puts you in (C:\Users\Me>), you have access to a source file or files you can build, debug and run.
This saves littering the world with Visual Studios File->New->Project->Console Application’s.

C# compiler setup

You’ll want to make sure you have the .net framework installed.

You’ll need to make sure you have your path variable included for csc.exe (the C# compiler).
There are several ways you can get to your path setup.
I normally just hit the keys [Windows] + [Pause-Break] combination, which brings up System Properties.

 

Go to the Advanced tab, and click the Environment Variables button.
I normally just enter the location of the compiler into the variables for the current user.
Select path, click edit.

 

I’m using .net 4. My path was C:\Windows\Microsoft.NET\Framework\v4.0.30319
If you already have other paths in the text field, past the same path in at the end of the line, but with a semi colon immediately before it.
Like so…

;C:\Windows\Microsoft.NET\Framework\v4.0.30319

If there is no existing path, add one by clicking the New button instead of the Edit button.
OK and apply all of that.
Now at your command prompt if you run path

C:\Users\Me>path

you will see all your path variables.
Now you can just add a file into C:\Users\Me\
This can be the file that you put your C# code into (I just called this “scratch”).

Once you think you’ve got your C# code compilable.
From the command prompt, type.

C:\Users\Me>csc scratch

That’s it, providing you have no errors your assembly is built.
If you’d like to know what’s in your assembly have a look here

Now if you want to debug with the likes of cordbg.exe, you’ll need the .NET SDK
You can find it here
At 600MB… I passed on this.
If you do decide to install the SDK, you’ll need to add the path, then you should be good to start using cordbg.exe.
Details on how to use cordbg.exe are here

I found the following links helpful.
http://www.johnsmiley.com/cis18/Smiley037.pdf
http://msdn.microsoft.com/en-us/library/ms379563%28VS.80%29.aspx
http://blogs.msdn.com/b/windowssdk/archive/2010/05/25/released-windows-sdk-for-windows-7-and-net-framework-4.aspx

Java compiler setup

If you haven’t already got it installed, you’ll need the Java Development Kit (JDK).
If you have the Java Runtime Environment (JRE) installed, this doesn’t mean you have the JDK.
If you have the JDK installed, you’ll have the JRE also.
In Windows you can see if you have these installed in Programs and Features in the Control Panel.

I got mine from here.
Once again open System Properties.
Go to the Advanced tab, and click the Environment Variables button.
Under the System variables, select Path, click edit.
As we did for csc.exe add the path with the semi colon in the front if there are already paths in there.

;E:\Program Files\Java\jdk1.6.0_18\bin

OK and apply all of that.
From then on, I just stick my java files in a directory in C:\Users\Me\
This way I just run cmd and cd into my dir that has my java files and I’m ready to go.

To compile your code into a bytecode file with a .class extension…

C:\Users\Me\MyDir>javac HelloWorld.java

To run your new bytecode…

C:\Users\Me\MyDir>java HelloWorld

Details here.

Now if you want to debug your java code on the command line,
it’s a breeze because the jdb.exe is in the same directory as java.exe and javac.exe.

 

All the commands and switches for jdb are here.

The Java documentation is very good, you won’t often not be able to find answers quickly.

There are also online options

Atlassian has some very comprehensive products that may suit.

Or you could go with a very light weight option.
compilr is a good one for C#, VB and Java.

Next post on this topic we’ll go into the language details.

Cya.

Garbage Collection in .NET 4.0

October 17, 2010

GC prior to .NET 4.0

Most of us know how the GC generations work, but I’ll give a very brief rundown for those that haven’t looked at them yet, or may need a refresher.
There are 3 of them. 0, 1, 2.
When an item is first created, it’s considered a gen 0 item.
If it survives subsequent collections (because it’s still in use), it becomes a gen 1 item and so on.
gen 0 items mainly consist of instance variables, where as the later generation items are generally of a more global scope.
gen 2 items often stick around for the duration of the application.

The GC operates in the following modes

server or workstation.

You can change the modes in either your app.config (that’s per app) or in the machine.config (that’s machine wide)
See here.

 

<Configuration>
	<runtime>
      <gcServer enabled="false" />
      <gcConcurrent enabled="true" />
   </runtime>
</Configuration>

As discussed by Tess Ferrandez (champion ASP.NET debugger)
The restrictions here are

  • you can not run the server version on a single proc box, it will default to workstation
  • you can not run concurrent while also running server
  • if the runtime is hosted, the hosts GC mode will override the configuration

workstation (the default mode), concurrent workstation (the default if >0 multi core procs exist),
Each of these modes are optimised for different scenarios.

Generally, it’s best not to change these settings, and especially on the machine level.
The .NET 4.0 GC is considerably smarter than previous versions of the framework, and there is even less reason to contemplate changing it.

Before .NET 4.0 the GC on a concurrent workstation could do most but not all of gen 0 and gen 1 reclaim at the same time as the gen 2 items.
The GC was unable to start additional collections when a current collection was in progress.
This meant that only memory in the current segment or block could be reclaimed.

In .NET 4.0 background GC has been introduced

This allows another GC (gen 0 and 1) to start at the same time as an existing full GC (gen 0, 1 and 2) is running.
This is able to significantly reduce the time a full GC takes.
Concurrent workstation GC has been replaced by background GC.
Server modes currently don’t support the background GC, but the CLR team is aiming to have this introduced in the next version of the framework.
GC in .NET 4.0 has been tested on machines with up to 128 CPU cores, with confirmed efficiency improvements.
Apparently they’ve reduced the time needed to suspend managed threads.

A couple of good informative links on background GC
http://blogs.msdn.com/b/ukadc/archive/2009/10/13/background-and-foreground-gc-in-net-4.aspx
http://channel9.msdn.com/shows/Going+Deep/Maoni-Stephens-and-Andrew-Pardoe-CLR-4-Inside-Background-GC/

Garbage Collection Notifications

In .NET 3.5SP1 onwards we’re provided with GC.RegisterForFullGCNotification.

Only available when concurrent GC is disabled (not the default).
Does not support concurrent garbage collection because memory allocations are allowed while concurrent garbage collection is in progress. See the <gcConcurrent> runtime setting for information about how to disable concurrent garbage collection.
Have a look at this example Microsoft has provided for Garbage Collection Notifications.
I’m not sure why MS has decided to implement notifications this way.
I would have thought a much better way to do this would be to simply subscribe to a notification event providing a delegate with WaitForFullGCProc (your conditional logic), rather than spin up a new thread which takes a ThreadStart delegate of WaitForFullGCProc (your conditional logic) and blocks on GC.WaitForFullGCApproach.

Thoughts / Feedback welcome.

LSP / DbC and .NET’s support

October 12, 2010
Part 2

Some Examples

Bear in mind, there are additional overloads for most/all of the following procedures.
Be sure to check them out.

Examples of Preconditions

Contract.Assert

Ensures that a condition is true when in debug mode.

 

Contract.Assert(MyInput != null);

Contract.Assume

Instructs code analysis tools to assume that the condition supplied as an argument is true, even if it can not be statically proven to always be true.
Good example of how this can be used.

 

Contract.Assume(MyInput != null);

Contract.Requires

Must be at the beginning of a method or property. Ensures that a condition is true before any other code.
The following example ensures that MyInput parameter is valid (not null in this case) before any other code is run on the same thread.

 

Contract.Requires(MyInput != null);

Contract.EndContractBlock

Is used to inform the compiler to treat all code before it as if it was a code contract.
This is useful for legacy code.
For example if you have parameter validation code that does not use Code Contract’s.
The contract tools recognise if/then/throw statements as preconditions when the statements appear first inside a method or property,
and the set of such statements are followed by a Contract method call, such as Requires, Ensures, EnsuresOnThrow or EndContractBlock.

 

if(MyInput == null)
    throw new NullReferenceException("The input is null");
Contract.EndContractBlock();

Examples of Postconditions

Contract.Ensures

Ensures that a condition is true on exit of a method or property.

 


Contract.Ensures(0 <= speed && speed <= 800);

Contract.EnsuresOnThrow

This method call must be at the beginning of a method or property, before any other code.
If an exception is thrown, the specified condition must be true and the exception being thrown is of the specified type parameter.

 

public bool AddAndProcessItem(T item)
{
	Contract.EnsuresOnThrow(!item.Added);

	try
	{
		 // add item to a collection and set it's Added property to true
	}
	catch (AdditionException)
	{
		 item.Added = false;
		 throw;
	}

	// ... other operations
	return true;
}

Contract.ForAll

Allows the iteration through a collection to ensure all members meet a specific requirement.

 

// first param MyCollection is of type IEnumerable
// second param is a delegate, or lambda (often called a predicate when used like this)
Contract.ForAll(MyCollection, x=>x!=null);

Object Invariants

Used to mark methods that contain object invariant specifications.
Defined by decorating a method with the ContractInvariantMethodAttribute.
ContractInvariantMethodAttribute can only be used on a single method per class.
The method must return void and carry out no operations other than a sequence of calls to Contract.Invariant

 

[ContractInvariantMethod]
void ObjectInvariant ()
{
   Contract. Invariant ( _y >= 0 );
   Contract. Invariant ( _x > _y );
   ...
}

Code Contract Values

There are also some useful pseudo variables that can be used when writing postconditions.

Contract.Result

A methods or properties return value can be refered to by Contract.Result<T>()
Contract.Result<T>() can be used only in the conditional expression for the Ensures contract.
Methods with a return type of void cannot refer to Contract. Result<T>() within their postconditions for the obvious reason.

 


Contract.Ensures(0 < Contract.Result());

Contract.OldValue

Contract.OldValue<T>(T value) can be used only in the conditional expression for the Ensures contract.
The generic type argument may be omitted whenever the compiler is able to infer its type.
There are a few restrictions around the use of OldValue. Have a look at the User Manual document. Listed at the end of the post.

 

Contract.Ensures(MyInput != Contract.OldValue(MyInput));

Pure

Using the PureAttribute indicates that a type, method, property, etc is pure, that is, It has no visible side-effects for callers.
You can only use a method/property/etc in a contract if it is declared Pure.
If the method/property/etc is not decorated with this attribute when called from within a contract, a warning will be given “Detected call to impure method”.

See this post, it has a good example.

Interface Contracts

Contracts for interfaces must be defined in a separate class decorated with the ContractClassForAttribute.
The interface sporting the contract must be decorated with the ContractClassAttribute specifying the type that has the interfaces contract.
The class that implements the interfaces contract is expected to be abstract.
The User Manual listed at the end of this post has some good examples of how to set up Interface Contracts.

 

[ContractClass(typeof(ContractForInteface))]
interface IDoSomething
{
	int DoSomething(int value);
}

[ContractClassFor(typeof(IDoSomething))]
sealed class ContractForInteface : IDoSomething
{
	int IDoSomething.DoSomething(int value)
	{
		Contract.Requires( value != 0);
		//contracts require a dummy value
		return default(int);
	}
}

There’s plenty of good documentation around Code Contracts

System.Diagnostics.Contracts Namespace
http://msdn.microsoft.com/en-us/library/system.diagnostics.contracts.aspx
The Contract Class
http://msdn.microsoft.com/en-us/library/system.diagnostics.contracts.contract.aspx
Brief description of Code Contracts and how to use them
http://msdn.microsoft.com/en-us/library/dd264808.aspx
The User Manual
http://download.microsoft.com/download/C/2/7/C2715F76-F56C-4D37-9231-EF8076B7EC13/userdoc.pdf
Code Contracts on DevLabs
http://msdn.microsoft.com/en-us/devlabs/dd491992.aspx
DimeCasts has a few web casts on Code Contracts
http://dimecasts.net/Casts/ByTag/Code%20Contracts

LSP / DbC and .NET’s support

October 11, 2010
Part 1

Two design principle’s I believe go hand in hand are

  1. Dr. Barbara Liskov’s Liskov Substitution Principle (LSP)
  2. Dr. Bertrand Meyer’s Design by Contract (DbC)

Lets keep our Object Oriented relationships adhering to the LSP and the DbC.
The LSP and the DbC provide both theory and practical guidelines of how we as software engineers can design…

  • Hierarchical components that behave the same across all layers.
  • Systems that produce predictable results on a layer, from changes made on another layer (providing architecture that can be reasoned about).
    As programmers, we need to be able to make correct assumptions about the behaviour of abstract data types and their derivatives.
    LSP and DbC provide us with the ability to create intuitive class hierarchies that support the object oriented architectural reasoning process.
  • Extensible code. Ability to add functionality without having to change existing code, but rather extending it.

LSP

The LSP states that you shouldn’t inherit from a base class unless the derived class truly “is a” more specific version of the base class (Liskov 1988).
Subtype objects must be behaviourally substitutable for supertype objects.
Programmers must be able to reason correctly about and rely upon the behaviour of subtypes using only the specification of the supertype behaviour.

Robert C. Martin wrote this excellent article on LSP.
Which I found here along with many other great articles.

Andy Hunt and Dave Thomas summarise LSP like this: “Subclasses must be usable through the base class interface without the need for the user to know the difference.”
In other words, all the routines defined in the base class should mean the same thing when they’re used in each of the derived classes.

As Steve McConnell in Code Complete puts it:
If you have a base class of Account and derived classes of CheckingAccount, SavingsAccount, and AutoLoanAccount.
A programmer should be able to invoke any of the routines derived from Account on any of Account’s subtypes without caring about which subtype a specific account object is.

If a program has been written so that the LSP is true, inheritance is a powerful tool for reducing complexity because a programmer can focus on the generic attributes of an object without worrying about the details.
If a programmer must be constantly thinking about semantic differences in a subclass implementation, then inheritance is increasing complexity rather than reducing it.
This goes against software’s primary technical imperative of managing complexity.
Suppose a programmer has to think this
“If I call the InterestRate() routine on CheckingAccount or SavingsAccount,
it returns the interest the bank pays,
but if I call InterestRate() on AutoLoanAccount I have to change the sign because it returns the interest the consumer pays to the bank.”
According to LSP, AutoLoanAccount should not inherit from the Account base class in this example
because the semantics of the InterestRate() routine are not the same as the semantics of the base class’s InterestRate() routine.

DbC

From my understanding. DbC provides constraints which enforce the LSP.
LSP is notional. DbC provides us with the practical approach.

.Net 4.0 gives us Code Contracts which directly support the DbC principle.
In looking at what the .Net framework 4.0 provides us with in the System.Diagnostics.Contracts namespace.
Bear in mind, this is not a language extension or a runtime extension. It’s just a library.
I think that using the static class’s in your code, that System.Diagnostics.Contracts provides would be a prime candidate for Aspect Oriented Programming.
It looks like the .Net CLR has support for AOP, although it doesn’t sound like it has been implemented very cleanly.
Have a look at this.
IMHO, when using AOP, the actual business code utilising the aspects, should not show any signs of the aspects (Code Contracts (in our case)).
Although it looks like there are some open source alternatives that do the job quite nicely.
Anyway, without diving into AOP, lets save it for another post and move on.

DbC is actually a registered trademark of Eiffel Software.
Bertrand Meyer when designing the Eiffel programming language came up with this term.
DbC provides us with class invariants, preconditions and postconditions.

C# for artists by Rick Miller has a good explanation of these.

Class Invariant

A class invariant is an assertion about an object property that must hold true for all valid states the object can assume.
For example, suppose an airplane object has a speed property that can be set to a range of integer values between 0 and 800.
This rule should be enforced for all valid states an airplane object can assume.
All methods that can be invoked on an airplane object must ensure they do not set the speed property to less than 0 or greater than 800.

Precondition

A precondition is an assertion about some condition that must be true before a method can be expected to perform it’s operation correctly.
For example, suppose the airplane object’s speed property can be incremented by some value and there exists in the set of airplane’s public interface methods one that increments the speed property any where from 1 to 5 depending on the value of the argument supplied to the method.
For this method to perform correctly, it must check that the argument is in fact a valid increment value of 1,2,3,4, or 5.
If the increment value tests valid then the precondition holds true and the increment method should perform correctly.
The precondition must be true before the method is called, therefore it is the responsibility of the caller to make the precondition true, and the responsibility of the called method to enforce the truth of the precondition.

Postcondition

A postcondition is an assertion that must hold true when a method completes its operations and returns to the caller.
For example, the airplane’s speed increment method should ensure that the class invariant speed property being 0<=speed<=800 holds true when the increment method completes its operations.

Code Contracts can run on .Net 4.0 or previous versions.

Code Contracts as a whole include

1)   Static methods from the Contract class

  • Express assumptions and constraints, I.E. the object invariants, pre and post conditions.

2)   Static analyzer or static checker

  • Provide compile-time analysis.
  • Checks for implicit contracts, such as null dereferences and checking array bounds, as well as the explicit developer provided contracts.
  • Only available in Microsoft’s Premium edition or above of Visual Studio.

3)   Runtime analyzer or binary rewriter

  • modifies the IL by injecting the contracts.
  • performs runtime checking.

Once you’ve installed either the Standard or Premium Edition of the managed code contracts from DevLabs.
Projects within Visual Studio will now have an extra property pane entitled “Code Contracts”.
This pane provides configuration options for both static and runtime contract checking.

I think the idea is that if your project is targeting:

  1. A .Net framework version prior to 4.0, the System.Diagnostics.Contracts namespace is in the Microsoft.Contracts.dll.
  2. The .Net framework version is 4.0, the System.Diagnostics.Contracts namespace is in the mscorlib.dll.

Distributed Version Control the solution?

October 3, 2010

Due to the fact that I am starting to need a Version Control System at home for my own work and the company I currently work for during the day could potentially benefit from a real Version Control System.

I’ve set out to do an R&D spike on what is available and would best suite the above mentioned needs.
I’ve looked at a large range of products available.

At this stage, due to my research and in talking to some highly regarded technical friends and other people about their experiences with different systems, I’ve narrowed them down to the following.

Subversion, Git and Mercurial (or hg)
Subversion is server based.
Git and hg are distributed (Distributed Version Control System (DVCS)).

The two types of VCS and some of their attributes.

Centralised (or traditional)

  • Is better than no version control.
  • Serves as a single backup.
  • Server maintenance can be time consuming and costly.
  • You should be able to be confident that the server has your latest changeset.

Distributed

  • Maintenance needs are significantly reduced, due to a number of reasons. One of which is… No central server is required.
  • Each peer’s working copy of the codebase is a complete clone.
  • There is no need to be connected to a central network. Which means users can work productively, even when network connectivity is unavailable.
  • Uses a peer-to-peer approach rather than a client-server approach that the likes of Subversion use.
  • Removes the need to rely on a single machine as a single point of failure.
    Although it is often a good idea to have a server that is always online and ready to accept changesets.
    As you don’t always know whether another peer has accepted all your changes or is online.
  • Most operations are much faster than the centralised model, as no network is involved.
  • Each copy of the repository effectively acts as a remote backup. Which has multiple benefits.
  • There is no canonical code base, only working copies.
  • Operations such as commits, viewing history and rolling back are fast, because there is no need to communicate with a central server.
  • A web of trust is used to merge code from disparate repositories.
  • Branching and Merging made easier.
  • No forced structure: a central server can be implemented or peers can control the codebase.
  • Although I don’t see huge benefits for a central server in my target scenario.
  • Buddy builds. A team member can pass a change set to another member to try before committing to a central location.
    This would stop broken CI builds.
  • There is a huge amount of flexibility with your layout.
  • With a well planned layout a Distributed Version Control System can do anything a Centralised system can do, with the additional benefit of easy merges.

In weighing up the pros and cons of distributed versus the centralised model.

I think for my target requirements,
a distributed system has more to offer in the way of time savings and hardware savings.
This page has a good explanation of the differences between Centralised and Distributed.
Here is a detailed list of comparisons of some of the more common systems.

Mercurial is ticking quite a few boxes for me.
Mercurial has a VisualStudio plug-in.
There is a GUI available for windows platforms and others that integrates Mercurial directly into your explorer.
It’s free, open, and being actively maintained.
Projects using Mercurial.

Mercurial is written in Python, which is another plus for me.
Binaries are freely available for Windows, GNU/Linux, Mac OS X, OpenSolaris.
The source is also available, so you can build it for most platforms.

Plenty of documentation here, plus the book.

Installation and Configuration. Covering Windows, Debian and more.
TortoiseHg has binaries for windows and debian, but only for Squeeze onwards by the look of it.
If your running Lenny, you can just use hg. apt-get install mercurial.
When I downloaded and installed the 64 bit version of TortoiseHg (v1.1.3 hg v1.6.3), it came with 4 comprehensive documents.

  1. Mercurial: The Definitive Guide 2010-02-21 as pdf
  2. TortoiseHg v1.1.3 Documentation in both pdf and chm
  3. Mercurial Command Reference

Very nice!
Turn off the indexing service on the working copies and repositories, and exclude them from virus scans
.
Can also get TortoiseHg here (For Debian, TortoiseHq isn’t available for Lenny).
Click the Tutorial link for the Quick start guide to TortoiseHg.

Once installed, start working through the following links.
http://tortoisehg.bitbucket.org/manual/1.1/quick.html
http://mercurial.aragost.com/kick-start/basic.html

Comments or thoughts?