Archive for the ‘UPS’ Category

Data Centre in a Rack

June 9, 2012

I recently took the plunge to install some of my more used networking components into a server rack.
I’d been putting this off for a few years.
Most of these components have been projects of mine which I’ve already blogged on in various places on this blog.
The obvious places are the following

There are also many other topics I’ve blogged on that form part of the work gone into these components and set-up of.
Check them out.

There’s also a home made router in an old $30 desktop pc run from a CF card.

Small Data Centre

Home Rack Server

HPR Pod cast on a bunch of good tools useful for setting up and maintaining an Open Source Data Centre.
ep0366 :: The Open Source Data Center

Questions welcome.
I’m happy to provide directions and insights from my experience.

Quick walk through, of my UPS library

August 4, 2011

Part three of a three part series

On setting up a UPS solution, to enable clean shutdown of vital network components.

In this post, we’ll be reviewing the library that performs the shutting down of our servers.

When I started on the library PowerOffUPSGuests.dll,
my thoughts were, if I’m going to do this, I wanted it to be extensible.
Able to shutdown pretty much any machines, requiring a clean shutdown due to power failure.
What I’ve done is left points to be easily extended in the future, when new requirements present themselves.

Source Code

The PowerOffUPSGuests repository is on bitbucket

I’m assuming you know how to use Mercurial and have it installed on your dev machine.
If you’re not familiar with Mercurial (hg) There’s a little here to get your feet wet.
Besides that, there is plenty of very good documentation on the net.
For starters, you’ll need to create a directory that you want to have as your repository.
For example, I use C:\Scripts.
Set this directory as a repository.
Then from within the directory,
issue an hg pull https://bitbucket.org/LethalDuck/poweroffupsguests
Then update your working directory to the tip of the local repository.

You’ll need a BinaryMist.PowerOffUPSGuests.dll.config in your <repository>\UPS\PowerOffUPSGuests\PowerOffUPSGuests\
which should look something like the following.
At this stage I’ve only been shutting down an ESXi host.
Replace the value at line 22 with the user that has privileges to perform shutdown on the server.
Replace the value at line 23 with the absolute path to the password file you’re about to generate.
Line 28 is the class name of the ServerController.
Line 34 denotes whether or not the Initiator will perform the shutdowns synchronously or asynchronously
The Server[n] and ServerPort[n] values that are commented out, are used when you want to intercept the messages being sent to/from the target server.
This is useful for examination and to help build the appropriate messages that the target server expects.

<?xml version="1.0"?>
<configuration>
   <assemblySettings>
      <!--As aditional target servers are added to the queue to be shutdown
         Keep the same nameing convention used below
         Just increment the suffix number for each target servers key.
         The first target suffix must start at 0.
         Additional target suffix's must be sequential.
         -->

      <!--Target servers to be shutdown-->

      <!--FreeNAS-->
      <!--add key="ServerUser0" value="YourUser"/>
      <add key="ServerUserPwFile0" value="Absolute directory that your password file resides\FileServerPw"/-->
      <!--add key="Server0" value="127.0.0.1"/--><!--localhost used for interception-->
      <!--add key="Server0" value="YourFileServerName"/-->
      <!--add key="ServerPort0" value="8080"/--><!--port used for interception-->
      <!--add key="ServerPort0" value="443"/-->
      <!--add key="Controller0" value="FreeNASController"/-->

      <!--ESXi-->
      <add key="ServerUser0" value="YourUser"/>
      <add key="ServerUserPwFile0" value="Absolute directory that your password file resides\VMHostPw"/>
      <!--add key="Server1" value="127.0.0.1"/--><!--localhost used for interception-->
      <add key="Server0" value="YourVSphereHostName"/>
      <!--add key="ServerPort1" value="8080"/--><!--port used for interception-->
      <add key="ServerPort0" value="443"/>
      <add key="Controller0" value="VMServerController"/>

      <!--Assembly settings-->

      <add key="LogFilePath" value="Some absolute path\Log.txt"/>
      <add key="CredentialEntropy" value="A set of comma seperated digits"/><!--"4,2,7,9,1" for example-->
      <add key="Synchronicity" value="Synchronous"/> <!--Check other values in Initiator.Synchronicity-->
      <add key="IgnoreSslErrors" value="true"/>
      <add key="Debug" value="true"/>

   </assemblySettings>
<startup><supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.0"/></startup></configuration>

Build the solution

You’ll now need to navigate to <repository>\UPS\PowerOffUPSGuests\
and run the file PowerOffUPSGuests.sln.
Build.
You should now notice a couple of binaries in <repository>\UPS\
along with the libraries config file.
As I mentioned in part two, the PCNS will execute <repository>\UPS\PowerOff.bat which will run PowerOffUPSGuests.ps1.
Which will inturn kick off the BinaryMist.PowerOffUPSGuests.dll which does the work.

Generate the encrypted password file

Just run the BinaryMist.PasswordFileCreator.exe.
This will provide the required user prompts to capture the password for the vSphere host you’re intending to shutdown.
Or if you would like to extend the project and create a specialized ServerController for your needs.
You can use the BinaryMist.PasswordFileCreator to capture any credentials and save to file.

The code that performs the capture and encryption looks like the following:


        static void Main() {
            CreatePasswordFile();
        }

        /// <summary>
        /// Provides interactive capture for the insertion of an encrypted password,
        /// based on the ServerUserPwFile0 specified in the BinaryMist.PowerOffUPSGuests.dll.config file.
        /// </summary>
        public static void CreatePasswordFile() {

            bool validPath;
            string path = null;
            string RetryMessage = "Please try again.";

            Console.WriteLine("You must create the password file running under" + Initiator.NewLine + "the same account that will run BinaryMist.PowerOffUPSGuests.");

            do {
                Console.WriteLine(
                    "From the BinaryMist.PowerOffUPSGuests.dll.config file." + Initiator.NewLine +
                    "Please specify the ServerUserPwFile[n] value" + Initiator.NewLine +
                    "for the encrypted Password to be stored to." + Initiator.NewLine +
                    "This must be a valid path" + Initiator.NewLine
                );

                try {
                    validPath = true;
                    path = Path.GetFullPath(Console.ReadLine());

                    if (!
                        ((IEnumerable<string>)ConfigReader.Read.AllKeyVals.Values)
                        .Contains<string>(
                            path, StringComparer.CurrentCultureIgnoreCase
                        )
                    ) {
                        Console.WriteLine(Initiator.NewLine);
                        Console.WriteLine("The value that was entered" + Initiator.NewLine +
                            "was not one of the specified values for ServerUserPwFile[n]");
                        Console.WriteLine(RetryMessage + Initiator.NewLine);
                        validPath = false;
                    }
                } catch (Exception) {
                    Console.WriteLine(Initiator.NewLine + "An invalid path was entered." + Initiator.NewLine + RetryMessage + Initiator.NewLine);
                    validPath = false;
                }
            } while (validPath == false);

            Console.WriteLine(
                Initiator.NewLine
                + "The password you are about to enter"
                + Initiator.NewLine
                + "will be encrypted to file \"{0}\""
                , path
            );

            byte[] encryptedBytes = ProtectedData.Protect(
                new ASCIIEncoding().GetBytes(pWord),
                ServerAdminDetails.CredentialEntropy(),
                DataProtectionScope.CurrentUser
            );
            File.WriteAllBytes(path, encryptedBytes);

            Console.WriteLine(
                Initiator.NewLine
                + Initiator.NewLine
                + string.Format(
                    "The password you just entered has been encrypted"
                    + Initiator.NewLine
                    + "and saved to {0}"
                    , path
                )
            );
            Console.WriteLine(Initiator.NewLine + "Press any key to exit");
            Console.ReadKey(true);
        }

 


        private static string pWord {
            get {
                bool passWordsMatch = false;
                bool firstAttempt = true;
                string passWord = null;
                while (!passWordsMatch) {

                    if (!firstAttempt)
                        Console.WriteLine(Initiator.NewLine + "The passwords did not match." + Initiator.NewLine);

                    Console.WriteLine(Initiator.NewLine + "Please enter the password..." + Initiator.NewLine);
                    string passWordFirstAttempt = GetPWordFromUser();
                    Console.WriteLine(Initiator.NewLine + Initiator.NewLine + "Please confirm by entering the password once again..." + Initiator.NewLine);
                    string passWordSecondAttempt = GetPWordFromUser();

                    if (string.Compare(passWordFirstAttempt, passWordSecondAttempt) == 0) {
                        passWordsMatch = true;
                        passWord = passWordFirstAttempt;
                    }
                    firstAttempt = false;
                }
                Console.WriteLine(Initiator.NewLine + Initiator.NewLine + "Success, the passwords match." + Initiator.NewLine);
                return passWord;
            }
        }

 


        private static string GetPWordFromUser() {
            string passWord = string.Empty;
            ConsoleKeyInfo info = Console.ReadKey(true);
            while (info.Key != ConsoleKey.Enter) {
                if (info.Key != ConsoleKey.Backspace) {
                    if (info.KeyChar < 0x20 || info.KeyChar > 0x7E) {
                        info = Console.ReadKey(true);
                        continue;
                    }
                    passWord += info.KeyChar;
                    Console.Write("*");
                    info = Console.ReadKey(true);
                } else if (info.Key == ConsoleKey.Backspace) {
                    if (!string.IsNullOrEmpty(passWord)) {
                        passWord = passWord.Substring
                            (0, passWord.Length - 1);
                        Console.SetCursorPosition(Console.CursorLeft - 1, Console.CursorTop);
                        Console.Write(' ');
                        Console.SetCursorPosition(Console.CursorLeft - 1, Console.CursorTop);
                    }
                    info = Console.ReadKey(true);
                }
            }
            return passWord;
        }

Once you’ve created the password file, you’re pretty much ready to start testing.
If you’ve followed the directions covered in the first two parts of this series, you should be good to go.
Part one.
Part two.

We’ll quickly go through some of the more interesting parts of the code…

The ConfigReader

The constructor loads the _settings IDictionary by calling the ReadConfig function.
.net libraries don’t usually contain an assembly config file.
This is how we get around it.
We read the name of the assemblies config file (see line 70).
We then load the configuration into an XmlDocument.
Create and populate the XmlNodeList.
Return the populated IDictionary.

    /// <summary>
    /// Reads the libraries configuration into memory.
    /// Provides convienient readonly access of the configuration from a single instance.
    /// </summary>
    public class ConfigReader {

        #region singleton initialization
        private static readonly Lazy<ConfigReader> _instance = new Lazy<ConfigReader>(() => new ConfigReader());

        /// <summary>
        /// constructor that sets the value of our "_settings" variable
        /// </summary>
        private ConfigReader() {
            _settings = ReadConfig(Assembly.GetCallingAssembly());
        }

        /// <summary>
        /// The first time Read is called must be from within this assembly,
        /// in order to create the instance of this class for the containing assembly.
        /// </summary>
        public static ConfigReader Read {
            get {
                return _instance.Value;
            }
        }
        #endregion

        /// <summary>
        /// settings to be used throughout the class
        /// </summary>
        private IDictionary _settings;
        /// <summary>
        /// constant name for the node name we're looking for
        /// </summary>
        private const string NodeName = "assemblySettings";

        /// <summary>
        /// class Indexer.
        /// Provides the value of the specified key in the config file.
        /// If the key doesn't exist, an empty string is returned.
        /// </summary>
        public string this[string key] {
            get {
                string settingValue = null;

                if (_settings != null) {
                    settingValue = _settings[key] as string;
                }

                return settingValue ?? string.Empty;
            }
        }

        public IDictionary<string, string> AllKeyVals {
            get {
                IDictionary<string, string> settings = new Dictionary<string, string>();
                foreach(DictionaryEntry item in _settings) {
                    settings.Add((string)item.Key, (string)item.Value);
                }
                return settings;
            }
        }

        /// <summary>
        /// Open and parse the config file for the provided assembly
        /// </summary>
        /// <param name="assembly">The assembly that has a config file.</param>
        /// <returns></returns>
        private static IDictionary ReadConfig(Assembly assembly) {
            try {
                string cfgFile = assembly.CodeBase + ".config";

                XmlDocument doc = new XmlDocument();
                doc.Load(new XmlTextReader(cfgFile));
                XmlNodeList nodes = doc.GetElementsByTagName(NodeName);

                foreach (XmlNode node in nodes) {
                    if (node.LocalName == NodeName) {
                        DictionarySectionHandler handler = new DictionarySectionHandler();
                        return (IDictionary)handler.Create(null, null, node);
                    }
                }
            } catch (Exception e) {
                Logger.Instance.Log(e.Message);
            }

            return (null);
        }

        #region Config related settings

        /// <summary>
        /// Readonly value, specifying whether debug is set to true in the assemblySettings of the BinaryMist.PowerOffUPSGuests.dll.config file.
        /// </summary>
        public bool Debug {
            get {
                if (_debug != null)
                    return _debug == true ? true : false;
                _debug = Read["Debug"] == "true" ? true : false;
                return _debug == true ? true : false;
            }
        }
        private bool? _debug;

        #endregion

    }

The Initiator


private void ShutdownSynchronously(Queue<ServerController> serverControllers) {
    foreach (ServerController serverController in serverControllers) {
        serverController.Shutdown();
    }
}

private void ShutdownAsynchronously(Queue<ServerController> serverControllers) {
    Action[] shutdownActions = new Action[serverControllers.Count];
    ServerController[] serverControllerArray = serverControllers.ToArray();

    for (int i = 0; i < serverControllerArray.Length; i++) {
        shutdownActions[i] = serverControllerArray[i].Shutdown;
    }

    try {
        Parallel.Invoke(shutdownActions);
    }
        // No exception is expected in this example, but if one is still thrown from a task,
        // it will be wrapped in AggregateException and propagated to the main thread. See MSDN example
    catch (AggregateException e) {
        Logger.Instance.Log(string.Format("An action has thrown an exception. THIS WAS UNEXPECTED.\n{0}", e.InnerException));
        throw new Exception();
    }
}

public string InitShutdownOfServers() {
    Logger.Instance.LogTrace();
    Queue<ServerController> serverControllers = new Queue<ServerController>();

    try {
        foreach (ServerAdminDetails serverAdminDetail in ServerAdminDetails.QueuedDetails) {

            Type t = Type.GetType(GetType().Namespace + "." + serverAdminDetail.ServerControllerType);
            serverControllers.Enqueue(Activator.CreateInstance(t, serverAdminDetail) as ServerController);
        }
    } catch(Exception e) {
        Logger.Instance.Log("Exception occured while enqueueing the server controllers. Details follow:" +
            NewLine +
            e.ToString()
        );
        throw;
    }

    bool ignoreCase = true;
    Synchronicity synchronicity = (Synchronicity)Enum.Parse(typeof (Synchronicity), ConfigReader.Read["Synchronicity"], ignoreCase);

    if(synchronicity == Synchronicity.Synchronous)
        ShutdownSynchronously(serverControllers);
    else
        ShutdownAsynchronously(serverControllers);

    return "InitShutdownOfServers successfully executed.";
}

In order to place the ServerController‘s on the serverController‘s Queue (line 33)… via the iterator provided by the Queue of ServerAdminDetails returned from
the static QueuedDetails property of ServerAdminDetails,
We must first instantiate the single instance of ServerAdminDetails,
which is what line 30 does.

In order for ServerAdminDetails to be constructed, the static _queue member (which is a Lazy of Queue of ServerAdminDetails) must be initialized first.
In order for the _queue member to be initialized with a Queue of ServerAdminDetails, the static QueueServersForShutdown procedure must be called.

This is where we pull out the values from the BinaryMist.PowerOffUPSGuests.dll.config shown above with the help of the ConfigReader.
As you can see, we iterate through the config file, building up the Queue of ServerAdminDetails  until we’ve read all the appropriate values.
Each pass through the loop instantiates a new ServerAdminDetails (non singleton because of inside class scope) with the values we pulled from the config file.

The ServerAdminDetails

    /// <summary>
    /// Provides the administration details of the servers listed in the BinaryMist.PowerOffUPSGuests.dll.config file.
    /// </summary>
    public class ServerAdminDetails {

        #region singleton initialization
        private static readonly Lazy<Queue<ServerAdminDetails>> _queue = new Lazy<Queue<ServerAdminDetails>>(QueueServersForShutdown);

        private ServerAdminDetails(string serverController, string serverName, string serverPort, string userName, byte[] serverCredential) {
            ServerControllerType = serverController;
            ServerName = serverName;
            ServerPort = serverPort;
            UserName = userName;
            Password = serverCredential;
        }

        /// <summary>
        /// Provides the process wide single instance queue of each servers admin details.
        /// </summary>
        public static Queue<ServerAdminDetails> QueuedDetails {
            get {
                return _queue.Value;
            }
        }
        #endregion

        private static Queue<ServerAdminDetails> ServersQueuedForShutdown { get; set; }

        private static byte[] GetMyCredential(string pWFileName) {
            try {
                return File.ReadAllBytes(pWFileName);
            } catch(Exception e) {
            string error = string.Format(
                "Error occured while instantiating a ServerAdminDetails instance for queueing. Specificaly while reading bytes from the following file: {0}{1}Exception details follow.{1}{2}",
                pWFileName,
                Initiator.NewLine,
                    e
                );
                Logger.Instance.Log(error);
                throw new Exception(error);
            }
        }

        private static Queue<ServerAdminDetails> QueueServersForShutdown() {

            ServersQueuedForShutdown = new Queue<ServerAdminDetails>();

            const int firstServerIndex = 0;
            int serverCount = firstServerIndex;
            string empty = string.Empty;

            do
            {
                string controller = ConfigReader.Read["Controller" + serverCount];
                string server = ConfigReader.Read["Server" + serverCount];
                string serverPort = ConfigReader.Read["ServerPort" + serverCount];
                string serverUser = ConfigReader.Read["ServerUser" + serverCount];
                string serverUserPwFile = ConfigReader.Read["ServerUserPwFile" + serverCount];

                if (controller == empty || server == empty || serverPort == empty || serverUser == empty || serverUserPwFile == empty)
                    break;

                ServersQueuedForShutdown.Enqueue(
                    new ServerAdminDetails(
                        controller,
                        server,
                        serverPort,
                        serverUser,
                        GetMyCredential(Path.GetFullPath(serverUserPwFile))
                    )
                );

                Logger.Instance.Log (
                    string.Format (
                        "Server admin details of Controller: {0}, Server: {1}, ServerPort: {2}, ServerUser: {3}, ServerUserPwFile: {4} added to queued element number {5}.",
                        controller,
                        server,
                        serverPort,
                        serverUser,
                        serverUserPwFile,
                        serverCount
                    )
                );
                serverCount++;

            } while (true);
            return ServersQueuedForShutdown;
        }

        /// <summary>
        /// Retreives the entropy found in the BinaryMist.PowerOffUPSGuests.dll.config file, used to encrypt all the passwords.
        /// </summary>
        /// <returns>byte[]</returns>
        public static byte[] CredentialEntropy() {
            string[] numbers = ConfigReader.Read["CredentialEntropy"].Split(',');
            byte[] entropy = new byte[numbers.Length];
            for (int i = 0; i < numbers.Length; i++) {
                entropy[i] = Byte.Parse(numbers[i]);
            }
            return entropy;
        }

        /// <summary>
        /// The name of the ServerController child type.
        /// </summary>
        internal string ServerControllerType { get; private set; }

        /// <summary>
        /// The name of the server
        /// </summary>
        internal string ServerName { get; private set; }

        /// <summary>
        /// The port of the server
        /// </summary>
        internal string ServerPort { get; private set; }

        /// <summary>
        /// The user name for the server
        /// </summary>
        internal string UserName { get; private set; }

        /// <summary>
        /// The password for the user
        /// </summary>
        internal byte[] Password { get; private set; }
    }

Back to line 33 of the Initiator.
Now that we can access each ServerAdminDetails within the Queue of ServerAdminDetails via the Queue iterator provided by ServerAdminDetails.QueuedDetails.
We can create the ServerController child types using the late bound Activator.CreateInstance method, based on the ServerAdminDetails.ServerControllerType.
A reference to each ServerController child instance is added to the serverControllers queue.
The Shutdown procedure for each ServerController child is then called.

The ServerController

Notice the constructor which has been called by the child’s constructor, calls back to the child’s AssembleRequests procedure before completing.

    /// <summary>
    /// Controls the process of shutting down the associated server.
    /// </summary>
    /// <remarks>
    /// An instance of this class is created indirectly via the more specific concrete creators for each server that requires shutdown.
    /// Plays the part of the Creator, in the Factory Method pattern.
    /// </remarks>
    internal abstract class ServerController {

        protected static readonly string NewLine = Initiator.NewLine;

        protected enum RequestMethod {
            Get,
            Post
        }

        /// <summary>
        /// Constructor for the <see cref="ServerController"/> class.
        /// Called via the more specific children to initialize the less specific members.
        /// </summary>
        /// <param name="serverAdminDetails">
        /// The details required to create the messages that need to be sent to the server in order to perform the shutdown.</param>
        public ServerController(ServerAdminDetails serverAdminDetails) {
            ServerAdminDetails = serverAdminDetails;
            RequestAssembler = new RequestAssembler();
            SoapEnvelopes = new Queue<XmlDocument>();
            this.AssembleRequests();
        }

        protected ServerAdminDetails ServerAdminDetails { get; set; }

        protected RequestAssembler RequestAssembler { get; set; }

        /// <summary>
        /// Reference a <see cref="System.Collections.Generic.Queue{System.Xml.XmlDocument}">queue</see> of soap envelopes.
        /// used by the children of this class to send to the server.
        /// </summary>
        public Queue<XmlDocument> SoapEnvelopes { get; protected set; }

        /// <summary>
        /// Initial preparation of messages that will be sent to the server to perform shutdown.
        /// </summary>
        /// <remarks>Factory method.</remarks>
        public abstract void AssembleRequests();

        /// <summary>
        /// Completes the compilation of the sequence of messages that need to be sent to the server in order to perform the shutdown.
        /// Sends the messages.
        /// </summary>
        public abstract void Shutdown();

        protected void NotifyOfShutdown() {
            Logger.Instance.Log(
                string.Format(
                    "{0}.Shutdown on server: {1} has now been executed.",
                    ServerAdminDetails.ServerControllerType,
                    ServerAdminDetails.ServerName
                )
            );
        }

        protected bool SimplePing() {
            string serverName = ServerAdminDetails.ServerName;
            Logger.Instance.Log(string.Format("Performing Ping test on server: {0}", serverName));
            Ping pingSender = new Ping();
            int pingRetry = 3;
            PingReply reply = null;
            for (int i = 0; i < pingRetry; i++) {
                try {
                    //may take a couple of tries, as may time out due to arp delay
                    pingSender.Send(serverName);
                    Logger.Instance.Log(string.Format("Initiating Ping number {0} of {1} on server: {2}. ", i + 1, pingRetry, serverName));
                    reply = pingSender.Send(serverName);

                    if (reply.Status == IPStatus.Success) break;

                } catch (Exception e) {
                    Logger.Instance.Log(e.ToString());
                }
            }

            bool optionsAvailable = reply.Options != null;
            string noOptions = "No Options available";

            Logger.Instance.Log(
                "Reply status for server: " + serverName + " was " + reply.Status + ". " + NewLine +
                "Address: " + reply.Address.ToString() + ". " + NewLine +
                "RoundTrip time: " + reply.RoundtripTime + ". " + NewLine +
                "Time to live: " + ((optionsAvailable) ? reply.Options.Ttl.ToString() : noOptions) + ". " + NewLine +
                "Don't fragment: " + ((optionsAvailable) ? reply.Options.DontFragment.ToString() : noOptions) + ". " + NewLine +
                "Buffer size: " + reply.Buffer.Length + ". "
                );

            return reply.Status == IPStatus.Success;
        }
    }

The ServerController children

The AssembleRequests constructs as much of the SOAP envelopes as it can,
without knowing all the information that the target server will provide to be able to complete the messages before being sent.
The RequestAssembler‘s CreateSoapEnvelope does the assembly of the SOAP envelope.
You’ll notice that the RequestAssembler‘s CreateLoginSoapEnvelope takes an extra argument.
The ServerAdminDetails is passed so that the target server’s credentials can be included in the SOAP envelope.
The SOAP envelopes are then queued ready for dispatch.

Now when line 02 or 11 of the Initiator is executed,
line 143 of the VMServerController will be called. That’s Shutdown.
Then we dequeue, send request, and process the response in DequeueSendRequestProcessResponse.
Passing in an optional parameter of Action of HttpWebResponse (the lambda).
From DequeueSendRequestProcessResponse,
we dequeue each SOAP envelope and call CreateWebRequest,
which in-turn, palms the work it knows about to the less specific RequestAssembler‘s CreateWebRequest as a lambda.
Now in DequeueSendRequestProcessResponse, when we get our initialized HttpWebRequest back,
We pass both the SOAP envelope and the HttpWebRequest to the RequestAssembler‘s InsertSoapEnvelopeIntoWebRequest‘s procedure to do the honors.

    /// <summary>
    /// Controls the process of shutting down the associated vSphere server.
    /// </summary>
    /// <remarks>
    /// Plays the part of the Concrete Creator, in the Factory Method pattern.
    /// </remarks>
    internal class VMServerController : ServerController {

        private static string _operationIDEndTag = "</operationID>";
        private static string _operationIDTags = "<operationID>" + _operationIDEndTag;
        private uint _operationIDaVal = 0xAC1CF80C;
        private uint _operationIDbVal = 0x00000000;
        private const int EstimatedHeaderSize = 45;

        private enum RequestType {
            Hello, HandShake, Login, Shutdown
        }

        private string _host;
        private string _uRL;
        private readonly string _action = @"""urn:internalvim25/4.1""";
        private readonly string _userAgent = @"VMware VI Client/4.0.0";
        private KeyValuePair<string, string> _cookie;

        public VMServerController(ServerAdminDetails serverAdminDetails) : base(serverAdminDetails) {

        }

        /// <summary>
        /// Loads the <see cref="ServerController.SoapEnvelopes">SoapEnvelopes</see> with the sequence of messages
        /// that need to be sent to the server in order to perform the shutdown.
        /// </summary>
        /// <remarks>
        /// Factory method implementation
        /// </remarks>
        public override void AssembleRequests() {
            Logger.Instance.LogTrace();
            _host = "https://" + ServerAdminDetails.ServerName + ":" +ServerAdminDetails.ServerPort;
            _uRL = "/sdk";
            SoapEnvelopes.Enqueue(CreateHelloEnvelope());
            SoapEnvelopes.Enqueue(CreateHandshakeEnvelope());
            SoapEnvelopes.Enqueue(CreateLoginEnvelope());
            SoapEnvelopes.Enqueue(CreateShutdownEnvelope());
        }

        private string InitialHeaderContent() {
            StringBuilder headerContent = new StringBuilder(_operationIDTags, EstimatedHeaderSize);
            headerContent.Insert(headerContent.ToString().IndexOf(_operationIDEndTag), _operationIDaVal.ToString("X8") + "-" + (++_operationIDbVal).ToString("X8"));
            return headerContent.ToString();
        }

        private XmlDocument CreateHelloEnvelope() {
            Logger.Instance.LogTrace();
            string bodyContent = @"
    <RetrieveServiceContent xmlns=""urn:internalvim25"">
      <_this xsi:type=""ManagedObjectReference"" type=""ServiceInstance"" serverGuid="""">ServiceInstance</_this>
    </RetrieveServiceContent>";
            return RequestAssembler.CreateSoapEnvelope(InitialHeaderContent(), bodyContent);
        }

        private XmlDocument CreateHandshakeEnvelope() {
            Logger.Instance.LogTrace();
            string bodyContent = @"
    <RetrieveInternalContent xmlns=""urn:internalvim25"">
      <_this xsi:type=""ManagedObjectReference"" type=""ServiceInstance"" serverGuid="""">ServiceInstance</_this>
    </RetrieveInternalContent>";
            return RequestAssembler.CreateSoapEnvelope(InitialHeaderContent(), bodyContent);
        }

        private XmlDocument CreateLoginEnvelope() {
            Logger.Instance.LogTrace();
            string bodyContent = @"
    <Login xmlns=""urn:internalvim25"">
      <_this xsi:type=""ManagedObjectReference"" type=""SessionManager"" serverGuid="""">ha-sessionmgr</_this>
      <userName></userName>
      <password></password>
      <locale>en_US</locale>
    </Login>";
            try {
                // As VMware insist on putting credentials in the SOAP body what else can we do?
                return RequestAssembler.CreateLoginSoapEnvelope(InitialHeaderContent(), bodyContent, ServerAdminDetails);
            } catch(InvalidCredentialException e) {
                string error = string.Format(
                    "Error occured during a call to CreateLoginSoapEnvelope, for server: {0}.{1}Exception details follow.{1}{2}",
                    ServerAdminDetails.ServerName,
                    NewLine,
                    e
                );
                Logger.Instance.Log(error);
                throw new Exception(error);
            }
        }

        private XmlDocument CreateShutdownEnvelope() {
            Logger.Instance.LogTrace();
            string bodyContent = @"
    <ShutdownHost_Task xmlns=""urn:internalvim25"">
      <_this xsi:type=""ManagedObjectReference"" type=""HostSystem"" serverGuid="""">ha-host</_this>
      <force>true</force>
    </ShutdownHost_Task>";
            return RequestAssembler.CreateSoapEnvelope(InitialHeaderContent(), bodyContent);
        }

        private HttpWebRequest CreateWebRequest(string uRL, KeyValuePair<string, string> cookie) {
            return RequestAssembler.CreateWebRequest(
                uRL,
                (request)=>{
                    request.Method = RequestMethod.Post.ToString();
                    request.UserAgent = _userAgent;
                    request.ContentType = "text/xml; charset=\"utf-8\"";
                    request.Headers.Add("SOAPAction", _action);
                    request.Accept = "text/xml";
                    request.KeepAlive = true;

                    if (!string.IsNullOrEmpty(cookie.Key))
                        request.Headers.Add("Cookie", cookie.Key + "=" + cookie.Value);
                }
            );
        }

        private void DequeueSendRequestProcessResponse(RequestType requestType, KeyValuePair<string, string> cookie, Action<HttpWebResponse> additionalResponseProcessing = null) {
            Logger.Instance.Log(string.Format("Will now attempt sending {0} message to server: {1}", requestType, ServerAdminDetails.ServerName));
            XmlDocument soapEnvelope = SoapEnvelopes.Dequeue();
            HttpWebRequest httpWebRequest = CreateWebRequest(_host + _uRL, cookie);
            RequestAssembler.InsertSoapEnvelopeIntoWebRequest(soapEnvelope, httpWebRequest);

            string soapResult;
            using (HttpWebResponse response = (HttpWebResponse)httpWebRequest.GetResponse())
            using (Stream responseStream = response.GetResponseStream())
            using (StreamReader streamReader = new StreamReader(responseStream)) {
                //pull out the bits we need for the next request.

                if (additionalResponseProcessing != null) {
                    additionalResponseProcessing(response);
                }

                soapResult = streamReader.ReadToEnd();
            }
        }

        /// <summary>
        /// Perform the shutdown of the server specified in the <see cref="ServerAdminDetails">server admin details</see>.
        /// </summary>
        public override void Shutdown() {

            bool serverOnline = SimplePing();
            Logger.Instance.Log(
                serverOnline
                    ? string.Format("Initiating sending of {0} to server: {1}", RequestType.Hello, ServerAdminDetails.ServerName)
                    : string.Format("Could not reach server: {0}. Aborting shutdown of server: {0}", ServerAdminDetails.ServerName)
            );

            ServicePointManager.ServerCertificateValidationCallback += ValidateRemoteCertificate;

            KeyValuePair<string, string> emptyCookie = new KeyValuePair<string, string>();
            DequeueSendRequestProcessResponse(
                RequestType.Hello,
                emptyCookie,
                (response)=> {
                    string[] setCookieElementsResponse = response.Headers["Set-Cookie"].Split(new[] { "\"" }, StringSplitOptions.RemoveEmptyEntries);
                    _cookie = new KeyValuePair<string, string>(setCookieElementsResponse[0].TrimEnd('='), "\"" + setCookieElementsResponse[1] + "\"");
                }
            );
            DequeueSendRequestProcessResponse(RequestType.HandShake, _cookie);
            DequeueSendRequestProcessResponse(RequestType.Login, _cookie);
            DequeueSendRequestProcessResponse(RequestType.Shutdown, _cookie);
            NotifyOfShutdown();
        }

        private static string RemoteCertificateDetails(X509Certificate certificate) {
            return string.Format(
                "Details of the certificate provided by the remote party are as follows:" + NewLine +
                "Subject: {0}" + NewLine +
                "Issuer: {1}",
                certificate.Subject,
                certificate.Issuer
            );
        }
    }

The RequestAssembler

    /// <summary>
    /// Provides ancillary operations for <see cref="ServerController">server controllers</see>
    /// that assist in the creation of the requests intended for dispatch to the servers requiring shutdown.
    /// </summary>
    internal class RequestAssembler {

        private static string _soapEnvelope = @"<soap:Envelope xmlns:xsd='http://www.w3.org/2001/XMLSchema' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns:soap='http://schemas.xmlsoap.org/soap/envelope/'>
    <soap:Header>
    </soap:Header>
    <soap:Body>
    </soap:Body>
</soap:Envelope>";

        /// <summary>
        /// Produces a SOAP envelope in XML form with the server admins user credentials
        /// </summary>
        /// <param name="headerContent">Content targeted for between the header tags.</param>
        /// <param name="bodyContent">Content targeted for between the body tags.</param>
        /// <param name="serverAdminDetails">The <see cref="ServerAdminDetails"/> instance used to access the user name and password.</param>
        /// <returns>The SOAP envelope.</returns>
        public XmlDocument CreateLoginSoapEnvelope(string headerContent, string bodyContent, ServerAdminDetails serverAdminDetails) {

            string failMessage = "The credentials were not correctly set. Username: {0} Password byte count: {1}";
            if (string.IsNullOrEmpty(serverAdminDetails.UserName) || serverAdminDetails.Password.Length < 1)
                throw new InvalidCredentialException(string.Format(failMessage, serverAdminDetails.UserName, serverAdminDetails.Password.Length));

            StringBuilder bodyContentBuilder= new StringBuilder(bodyContent, bodyContent.Length + 10);
            bodyContentBuilder.Insert(
                bodyContentBuilder.ToString().IndexOf("</userName>"),
                serverAdminDetails.UserName
            );

            bodyContentBuilder.Insert(
                bodyContentBuilder.ToString().IndexOf("</password>"),
                decryptedCredential(serverAdminDetails)
            );

            return CreateSoapEnvelope(headerContent, bodyContentBuilder.ToString());
        }

        private string decryptedCredential(ServerAdminDetails serverAdminDetails) {
            try {
                return new ASCIIEncoding().GetString(
                    ProtectedData.Unprotect(
                        serverAdminDetails.Password,
                        ServerAdminDetails.CredentialEntropy(),
                        DataProtectionScope.CurrentUser
                    )
                );
            } catch(CryptographicException e) {

                // Retrieve the exception that caused the current
                // CryptographicException exception.
                Exception innerException = e.InnerException;
                string innerExceptionMessage = "";
                if (innerException != null) {
                    innerExceptionMessage = innerException.ToString();
                }

                // Retrieve the message that describes the exception.
                string message = e.Message;

                // Retrieve the name of the application that caused the exception.
                string exceptionSource = e.Source;

                // Retrieve the call stack at the time the exception occured.
                string stackTrace = e.StackTrace;

                // Retrieve the method that threw the exception.
                System.Reflection.MethodBase targetSite = e.TargetSite;
                string siteName = targetSite.Name;

                // Retrieve the entire exception as a single string.
                string entireException = e.ToString();

                // Get the root exception that caused the current
                // CryptographicException exception.
                Exception baseException = e.GetBaseException();
                string baseExceptionMessage = "";
                if (baseException != null) {
                    baseExceptionMessage = baseException.Message;
                }

                Logger.Instance.Log(
                    "Caught an unexpected exception:" + Initiator.NewLine
                    + entireException + Initiator.NewLine
                    + Initiator.NewLine
                    + "Properties of the exception are as follows:" + Initiator.NewLine
                    + "Message: " + message + Initiator.NewLine
                    + "Source: " + exceptionSource + Initiator.NewLine
                    + "Stack trace: " + stackTrace + Initiator.NewLine
                    + "Target site's name: " + siteName + Initiator.NewLine
                    + "Base exception message: " + baseExceptionMessage + Initiator.NewLine
                    + "Inner exception message: " + innerExceptionMessage + Initiator.NewLine
                );
                throw;
            }
        }

        /// <summary>
        /// Produces a SOAP envelope in XML form.
        /// </summary>
        /// <param name="headerContent">Content targeted for between the header tags.</param>
        /// <param name="bodyContent">Content targeted for between the body tags.</param>
        /// <returns><see cref="System.Xml.XmlDocument">The SOAP envelope</see>.</returns>
        public XmlDocument CreateSoapEnvelope(string headerContent, string bodyContent) {

            StringBuilder sb = new StringBuilder(_soapEnvelope);

            try {
                sb.Insert(sb.ToString().IndexOf(Initiator.NewLine + "    " + "</soap:Header>"), headerContent);
                sb.Insert(sb.ToString().IndexOf(Initiator.NewLine + "    " + "</soap:Body>"), bodyContent);
            } catch(Exception e) {
                Logger.Instance.Log(e.ToString());
                throw;
            }

            XmlDocument soapEnvelopeXml = new XmlDocument();
            soapEnvelopeXml.LoadXml(sb.ToString());

            return soapEnvelopeXml;
        }

        /// <summary>
        /// Creates a web request based on the url passed in.
        /// </summary>
        /// <param name="url">The target URL of the server that will be shutdown.</param>
        /// <param name="additionalWebRequestManipulation">delegate of type
        /// <see cref="System.Action{HttpWebRequest}">Action{HttpWebRequest}</see>.
        /// This is used to take additional tasking defined in the calling procedure.
        /// This procedure creates the web request,
        /// and passes it into this parameter for the additional initialization work to be performed on the web request.
        /// </param>
        /// <returns>
        /// An initialized <see cref="System.Net.HttpWebRequest">web request</see>,
        /// ready to have a <see cref="System.Xml.XmlDocument">soap envelope</see> inserted.</returns>
        public HttpWebRequest CreateWebRequest(string url, Action<HttpWebRequest> additionalWebRequestManipulation = null) {
            HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(url);

            if(additionalWebRequestManipulation != null) {
                additionalWebRequestManipulation(webRequest);
            }
            return webRequest;
        }

        /// <summary>
        /// Insert the <see cref="System.Xml.XmlDocument">soap envelope</see> into the <see cref="System.Net.HttpWebRequest">web request</see>.
        /// </summary>
        /// <param name="soapEnvelopeXml">
        /// The <see cref="System.Xml.XmlDocument">soap envelope</see> to be inserted into the
        /// <see cref="System.Net.HttpWebRequest">web request</see>.
        /// </param>
        /// <param name="webRequest">The <see cref="System.Net.HttpWebRequest">web request</see> that the
        /// <see cref="System.Xml.XmlDocument"/>soap envelope</see> is inserted into.
        /// </param>
        public void InsertSoapEnvelopeIntoWebRequest(XmlDocument soapEnvelopeXml, HttpWebRequest webRequest) {
            using (Stream stream = webRequest.GetRequestStream()) {
                soapEnvelopeXml.Save(stream);
            }
        }

        private static void InsertByteArrayIntoWebRequest(byte[] postData, HttpWebRequest webRequest) {
            using (Stream stream = webRequest.GetRequestStream()) {
                stream.Write(postData, 0, postData.Length);
            }
            webRequest.ContentLength = postData.Length;
        }

        /// <summary>
        /// Inserts the credentials from the <see cref="ServerAdminDetails">server admmin details</see> into the
        /// <see cref="System.Net.HttpWebRequest">web request</see> supplied.
        /// </summary>
        /// <param name="serverAdminDetails">
        /// The <see cref="ServerAdminDetails">server admin details</see>
        /// containing the information for the server that the request will be sent to.
        /// </param>
        /// <param name="webRequest">
        /// The <see cref="System.Net.HttpWebRequest">web request</see> that will have the server administration credentials inserted.
        /// </param>
        public void InsertCredentialsIntoWebRequest(ServerAdminDetails serverAdminDetails, HttpWebRequest webRequest) {
            InsertByteArrayIntoWebRequest(
                new ASCIIEncoding().GetBytes("username=" + serverAdminDetails.UserName + "&password=" + decryptedCredential(serverAdminDetails)),
                webRequest
             );
        }
    }


Let me know if any of this is unclear, and requires additional explanation.

Once again, the full source code can be found here.

Preparing APC Smart-UPS 1500 clients

July 26, 2011

Part two of a three part series

on Setting up a UPS solution, to enable clean shutdown of vital network components.

This post is about setting up the software that will be responsible for cleanly shutting down servers and workstations.

We have to decide which machine/s is/are going to be used to launch our script (which in turn is run by what APC call a command file).

Currently I’ve got an old laptop I pulled out of the rubbish about 5 years ago, with Windows XP running on it.
It’s got just enough battery capacity to stay alive for long enough to receive the event from the NMC (Network Management Card) and run my .dll that issues the shutdown sequence.
A couple of EeePC 901’s have also recently been made redundant, and I may use one of those with Windows 7 installed at some stage.
Currently all of my workstations and servers that don’t have batteries, I.E. notebooks are VM’s running on ESXi.
Oh… or servers that have their entire file system loaded into volatile memory, so if they are powered off, I.E. cold shutdown, there is no possible corruption of the file system.
What you can also do is host the PCNS (PowerChute Network Shutdown) on a VM, because once the shutdown of ESXi has been initiated, there is no stopping the sequence, and the VM’s will all be cleanly shutdown.
Or better still, use more than one machine to host PCNS, as they will operate on a first in first served basis.
As you’ll see here the NMC’s firmware and PCNS are quite extensible.
The above document is recommended reading if your planning on setting up an APC UPS and want to automate clean shutdowns.
Without reading, the comms can get a little confusing.

Setting up PCNS

Install the PowerChute Network Shutdown service

You can get a copy of v2.2.3 here
I later found out that there were later versions:
v2.2.4 linked to from here, which has additional documentation.
v3.0.0 linked to from here, which has additional documentation.
Both of which were linked to from here, which has additional manuals etc.
You can find the installation guide here.
The PCNS service needs to be run as a local Administrator as the default Local System account doesn’t have sufficient rights.
In saying all that, William Tournas from APC recommended I use PCNS 2.2.1 for Windows XP.
Additional 2.2.1 resources are found here.

If using vMA with PCNS 3.0, you go through a Web UI configuration wizard once installed.
If using PCNS with Windows, the configuration is part of the install.

Either way, the steps will look similar to the following:

netstat -a

Should show that PCNS is listening on TCP and UDP ports 3052

If it’s not, you’ll need to open those ports on your firewall.

If you’re looking at using a Linux based VM to host PCNS,
VMware provide vMA (vSphere Management Assistant) a CentOS VM image.
You can get the binary here.
You’ll also need PCNS.
Take your pick of the following binaries:
2.2.4
3.0.0
Along with the documentation:
PowerChute_NetworkShutdownv2.4-ReleaseNotes.htm
PowerChute_NetworkShutdownv3.0-ReleaseNotes.htm
You’ll have to have the same ports open, as PCNS will be listening on them.
A listing of iptables for the filter (default unless otherwise specified) table should look like the following:

For an easier to read output, try the following:

sudo iptables  -L -v -n --line-numbers | column -t

Once again, if these ports are not open, you’ll have to find which script is being used to set up the rules.
I’m not sure about CentOS, but in a Debian based system, you would normally put the firewall init script in /etc/init.d/
This script would call a script that sets up the rules and one that tears them down.
I’m going to be making a post about how I set up my firewall (iptables) rules for the netfilter module on our notebooks at some stage soon.
If I haven’t already done this and you need more help, just sing out.

I found the following links quite helpful with the setup:

Link1

ESXi.pdf linked to from here linked to from here.

This has a list of the ports that are supposed to be open on pcns 2.2.3 with ESX 3.5
I think this also applies to PCNS 3.0 and ESXi 4.1 which I tried out.

Also be aware that there’s a known issue with special characters in the credentials for PCNS 3.0

I read somewhere that the PCNS needs to have the same credentials as the NMC, so just be aware of this.

Could be useful for trouble shooting vMA (vSphere Management Assistant)
I made a couple of posts there.

PowerChute Network Shutdown v3.0 – Release Notes
goes through a whole lot of issues and work-arounds with PowerChute.
For example, discuss’s the correct way to run the command file, PowerOff.bat in our case.

The APC PCNS receives an event from the AP9606 (that’s the NMC (Network Management Card)) fitted to the UPS.
The script is launched by APC PCNS from a Windows or Linux box.
I read that PCNS will always shutdown the windows machine it’s running on.
This is not true.

My attempt at using a PowerShell script utilizing mostly VMware’s cmdlets to shutdown ESXi

PowerChute has an option to ‘run this command’ but it’s limited to 8.3 paths and won’t accept command line parameters.
A separate batch file is needed (I called it poweroff.bat)
that runs the shutdown script with the parameters – but that could shut down other ESXi boxes as well if required.

I was keen to use PowerShell to perform the shutdowns, as I’d read it was quite capable and also VMware supplied a large set of management cmdlets.

Install PowerCLI from here.
read the installation guide.
As an admin, run the following:

set-executionpolicy remotesigned

Details of executionpolicy here.

If running PowerOffUPSGuests.ps1 from command shell rather than from a batch file.
You need to add the PowerCLI snapin.

PS C:\scripts&gt; Add-PSSnapin VMware.VimAutomation.Core
PS C:\scripts&gt; . .\PowerOffUPSGuests.ps1 MyESXiHostName AdminUserName

This will establish the SSL connection to MyESXiHostName

Following are the PowerShell scripts I used.

First we had to create our password file to use to log in to vSphere.
See this post for how this was done.

PowerOffUPSGuests.bat (the command file)

echo off
REM VMware would have used the Export-Console cmdlet to export the name of the PowerShell snap-in PowerCLI uses.
REM to the PowerShell console file (.psc1)

REM Invoke the command with the call operator (The ampersand).
PowerShell.exe -PSConsoleFile "C:\Program Files\VMware\Infrastructure\vSphere PowerCLI\vim.psc1" "& "C:\Scripts\PowerOffUPSGuests.ps1" MyESXiServer.MyDomain MyUser

PowerOffUPSGuests.ps1 (the script that was going to do the work)

param ( [parameter(Mandatory=$true)][string] $vSphereServername,
   [parameter(Mandatory=$true)][string] $user
)

$HostCredential = C:\Scripts\Get-myCredential.ps1 $user C:\Scripts\mp.txt

Set-StrictMode -Version 2.0
Write-Host "Establishing connection to $vSphereServername" -ForegroundColor Yellow
Connect-VIServer -Server $vSphereServername -Protocol https -Credential $HostCredential

function Stop-VMOnVMHost {
   Write-Host "Shutting down guests." -ForegroundColor Yellow

   $vM = Get-VM | Where-Object {$_.PowerState -eq "PoweredOn" -and $_.Guest.State -eq "Running"}
   Write-Host "Shutting down the following guests: $vM " -ForegroundColor Yellow
   $vM | Shutdown-VMGuest -Confirm:$False
   $seconds = 300
   Write-Host "Waiting $seconds Seconds. "
   Start-Sleep -Seconds $seconds

   $vM = Get-VM | Where-Object {$_.PowerState -eq "PoweredOn"}
   Write-Host "Stopping the following guests: $vM " -ForegroundColor Yellow
   $vM | Stop-VM -RunAsync -Confirm:$False
   $seconds = 60
   Write-Host "Waiting %seconds Seconds. "
   Start-Sleep -Seconds $seconds
}

function Stop-VMHost {
   Write-Host "Setting state of $vSphereServername to maintenance mode. " -ForegroundColor Yellow
   Get-VMHost | ForEach-Object {
      $hostName = $_.Name
      Write-Host "Putting $hostName into maintenance mode. "
      Set-VMHost -vmhost $_ -state maintenance
      Write-Host "Stopping $hostName. "
      Stop-VMHost -vmhost $_ -RunAsync
   }
}

Stop-VMOnVMHost
Stop-VMHost
Write-Host "Shutdown Complete" -ForegroundColor Yellow

Tried my script and got the following:

Shutdown-VMGuest     Operation “Shutdown VM guest.” failed for VM “MyGuestNameHere” for the following reason: fault.Restriction.summary

I had a hunch that it was due to the read only restriction I had heard about.
So tried command straight from PowerShell console…
same result.
More details here.
PowerCLI references to shutting down ESX
http://pastebin.com/HgsbSpb7
http://www.sheenaustin.com/2011/02/20/vmware-ups-shutdown-script/
http://communities.vmware.com/message/1555286
http://spininfo.homelinux.com/news/vSphere_PowerCLI/2010/06/18/Shutdown_infrastructure_on_power_outage
http://blogs.vmware.com/kb/2010/09/managing-esxi-made-easy-using-powercli.html
http://www.vmware.com/support/developer/PowerCLI/PowerCLI41U1/html/index.html

So as it turned out, VMware has removed write access from PowerCLI to ESXi, in 4.0 onwards I think.

Back to Scripting SOAP

As I was kind of out of luck with using PowerCLI cmdlets,
I decided to write my own library,
that I would execute using PowerShell.

First command needs to shutdown my fileserver.
Issue:
hello, authenticate, shutdown

Used Burp suite to diagnose the http frames being sent received from/to vSphere client/ESXi.
I haven’t used this tool before, but it gave very good visibility of the messages being sent/received.
The vSphere client has a config file here:
C:\Program Files\VMware\Infrastructure\Virtual Infrastructure Client\Launcher\VpxClient.exe.config
that you can change the ports that the vSphere client sends/receives on,
but I found it easier to just set the IP address / Name field in the GUI
to point to 127.0.0.1:8080 This is where the Burp proxy listens on by default.

In Burp, you will also need to add another proxy listener to the proxy->options tab.
Set Local Listener to 8080,
Uncheck listen on loopback interface only,
Check support invisible proxying for non-proxy-aware clients.
The in-app help has good documentation on this.
Set redirect to host to the ESXi host name.
Set redirect to port to the ESXi’s default SSL port of 443
Select the generate CA-signed per-host certificates radio button.

I also made sure the new proxy rule was the only one running.
When Burp captures each frame, you can forward each one onto any one of the other tools in the suite.
This is a really nice tool.

My PowerOffUPSGuests.ps1 was about to significantly change too.
Running my PowerOffUPSGuests.ps1 script using PowerShell

PS C:\Scripts\UPS&gt; . ".\PowerOffUPSGuests.ps1"

We no longer need to pass any arguments to PowerOffUPSGuests.ps1

I was going to be using .net 4 libraries in my PowerOffUPSGuests.dll,
so needed to Let PowerShell know about the .net 4 CLR.
By default PS 2.0 is only aware of the .net 2.0 framework.

Some insight on this:
http://stackoverflow.com/questions/2094694/launch-PowerShell-under-net-4
http://tfl09.blogspot.com/2010/08/using-newer-versions-of-net-with.html
http://www.powergui.org/thread.jspa?threadID=13403

So needed to create a couple of config files for
%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\PowerShell.exe
and
PowerShell_ise.exe
with config appended
%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\PowerShell.exe.config
and
PowerShell_ise.exe.config
with the following contents:

<?xml version="1.0"?>
<configuration>
  <startup useLegacyV2RuntimeActivationPolicy="true">
     <supportedRuntime version="v4.0.30319"/>
     <supportedRuntime version="v2.0.50727"/>
   </startup>
</configuration>


This works for PowerShell, but not for PowerGUI (obviously) which I was using for debugging.


So If you still need PowerGUI you’ll have to add the registry hacks explained in the links above.
Remember to remove them once finished as they take affect system wide.

I also had some trouble with later versions of C# than 2.0 when compiling on the fly in PowerShell.
Although I was specifying the language.

Add-Type -Path $typePath -CompilerParameters $compilerParameters -Language csharpversion3


Found a workaround this bug here.

# add the block of code we call into
$code = [io.file]::ReadAllText((Join-Path -Path $scriptPath -ChildPath $powerOffUPSGuestsFile))
Add-Type $code -CompilerParameters $compilerParameters -Language CSharpVersion3


We’ll go over the library code in the third part of this series.

As it stands now, the C:\Scripts\UPS\PowerOff.bat looks like this

echo.
echo PowerOffUPSGuests.ps1 started at the following time: %time% &gt;&gt; C:\Scripts\UPS\Log.txt
"C:\WINDOWS\system32\WindowsPowerShell\v1.0\PowerShell.exe" C:\Scripts\UPS\PowerOffUPSGuests.ps1
echo PowerOffUPSGuests.ps1 finished at the following time: %time% &gt;&gt; C:\Scripts\UPS\Log.txt
echo.


The PowerOffUPSGuests.ps1 looks like this

Set-StrictMode -Version 2.0

# add the assembly that does the work.
Add-Type -Path C:\Scripts\UPS\PowerOffUPSGuests.dll

# instantiate a PowerOffUPSGuests instance
$powerOffUPSGuestsInstance = New-Object -TypeName BinaryMist.Networking.Infrastructure.PowerOffUPSGuests

Write-Host $powerOffUPSGuestsInstance.InitShutdownOfServers() -ForegroundColor Green

The essential files

Testing that everything works

I was unsure whether we were going to be able to get ESXi to cleanly shutdown it’s guest VM’s.
As I’d had some trouble with this previously.

I was thinking about writing a WCF service and client to shutdown windows guests for now.
The service being on the machine that needed to be cleanly shutdown.
Could use something like the following command line in the service.

shutdown.exe -m //MachineNeedingShutdown -t 10 -c "Shutting down due to UPS running on battery." -s -f

Wrapped in something like this…

Process shutdownMyBox = new Process();
shutdownMyBox.StartInfo.FileName = shutdown.exe;
shutdownMyBox.StartInfo.Arguments = "-m //MachineNeedingShutdown -t 10 -c \"Shutting down due to UPS running on battery.\" -s -f";
shutdownMyBox.Start();

I was sure there was a better way though.

The sequence of events I was thinking of was something like the following:

First we try to shutdown every VM guest, set vMGuestTimmer
If all VM guests shutdown
——try put host into maintenance mode, set timer.
——when in maintenance mode
———shutdown host
——if enter maintenance mode not successful within time set
———shutdown host
On vMGuestTimmer
——shutdown host

There was a better/easier way though

In the PCNS Web UI -> PowerChute->MachineName->Configure Events
You can set PowerOff.bat to run after 30 seconds, or for testing,
set it to something really small, so it runs the command file sooner.
Set the time that’s required for the command file to complete to 5 minutes.
Although I don’t think it matters that much, as long as there’s enough time to start the execution of the PowerShell script.
Once the script is running, we don’t care how long PCNS thinks it should wait, as it’s non blocking.

To test that pcns will run your batch file:
Just put some temporary script, something like the following in your
C:\Scripts\UPS\PowerOff.bat

time/T &gt;&gt; C:\Scripts\UPS\MyTest.txt

These are some links from APC to help get your PCNS command file running:
http://jpaa-en.apc.com/app/answers/detail/a_id/7712
http://jpaa-en.apc.com/app/answers/detail/a_id/2441
http://jpaa-en.apc.com/app/answers/detail/a_id/1175


What is needed for ESXi to shut down all machines cleanly?

Graceful shutdown work around for ESXi guests.

Also it’s important to make sure the root user of ESXi has the Administrator Role.

What is needed for Windows VM’s to shutdown cleanly?

First ascertain whether or not your VM is/isn’t being shutdown cleanly.
eventvwr is your friend.

The scripts that may play a part in the shutting down of the Windows VM’s.
If you have a look at the VMware Tools Properties->Scripts tab
You can see for the shutdown script, that it actually does nothing.
If you find that your Windows box is not shutting down cleanly…
Add a custom script to the “Shut Down Guest Operating System” Script Event
I just created a shutdown.bat with the following in it.

C:\Windows\System32\shutdown.exe -s -t 1

This cleared up any errors I was getting in my Windows7 logs.

What is needed for Linux VM’s to shutdown cleanly?

If you’re looking at Debian based systems…
View the relevant log that contains shutdown info.

sudo vi /var/log/messages

and

sudo vi /var/log/syslog

From command mode (that’s [Esc]) to show line numbers,

type

:set number

or

:set nu

To find the matches for “shutdown” (without quotes) ignoring case

sudo grep -i -n "shutdown" /var/log/messages

Or easier still…
Once the file’s open in vi,
From command mode

/shutdown

[n]            will repeat the search forward
[N]            will repeat the search in opposite direction

My Debian wheezy server wasn’t getting shutdown cleanly.
So tried to install vmware tools, but found the easier way was to use open-vm-tools
Added contrib to my /etc/apt/sources.list
Installed open-vm-tools open-vm-source
Had some trouble with the NZ repo for those packages, they were corrupt.
So renamed /etc/apt/apt.conf so apt-get wasn’t using my cached packages from apt-cacher.

sudo apt-get clean
sudo apt-get update
sudo apt-get install open-vm-tools open-vm-source

The scripts that may play a part in the shutting down of the Linux VM’s.
Read this link.
There are also vmware-tools scripts
Mine didn’t appear to do much, but my server was being shutdown cleanly now.


Shout out if anythings unclear.

In part three I’ll be going over the library I’ve written that actually does the work 😉

Preparing APC Smart-UPS 1500 for Critical Servers

June 16, 2011

Part one of a three part series

on Setting up a UPS solution, to enable clean shutdown of vital network components.

This post is essentially about setting up a Smart-UPS and it’s NMC (Network Management Card),
as the project I embarked upon was a little large for a single post.

Christchurch NZ used to have quite stable power,
but recent earthquakes we’ve been having have changed that.
Now we endure very unstable power.
This fact,
along with the fact that if my RAID arrays were being written to when a power outage occurred,
prompted me to get my A into G on this project.

For a while now I’ve been looking into setting up a UPS solution to support my critical servers.
I already had a couple of UPS’s
Liebert PowerSure 250 VA
Eaton Powerware 5110 500 VA
Both of which were a bit small to support a fairly hungry hypervisor, dedicated file server, 24 port Cisco catalyst switch and a home made router.
Also the FreeNAS (BSD ) driver for USB that was supposed to work with the 5110, didn’t seem to.
In considering the above; I had a couple of options.
With ESXi we can use an APC UPS and a network management card or the Powerware 5110 connected to a network USB hub
and a virtual guest listening to its events, ready to issue shutdown procedures as per James Pearce’s solution
but to any number of machines.

What I wanted was a single UPS plugged into a single box that would receive on battery events and do the work of shutting down the various machines listed (any type of machine, including virtual hosts and guests).
There didn’t appear to be a single piece of software that would do this, so I wrote it.
I’ll go over this in a latter post.

So I would need either a network connected USB hub. As explained here.

Simple Two Port Network Connected USB Hub

Hardware solutions and all work well from VM guests from what I’ve read.
AnywhereUSB from Digi …
USB server from Keyspan
USB Anywhere from Belkin

Software solutions, Need physical PC that has USB device/s plugged in.
USB@nywhere
USB over Network
USB Redirector

Or a network management card (something like the AP9606) for the UPS as explained here by James
Powerware 5110 doesn’t support a network management card, so only option I see for this UPS is a network connected USB hub.
APC SMART-UPS supports network management cards and I think these would be the best option for this UPS.

The AP9606

It was starting to look like an APC UPS would be the better option.
I had already been looking for one of these for quite a while, and I missed a couple of them.

The one I eventually picked up

APC Smart-UPS

Second hand APC Smart-UPS 1500
$200 + shipping = just under $300.
AP9606 NMC $50 + shipping = aprx $80.
So for $380 even if I needed a new battery ($250),
I still had a $1300 UPS, NMC not included, for $600.
Turned out the battery was fine,
so all up $380 to support a bunch of hardware.

You’ll need to give the card an IPv4 address that suites your subnet.
As my card was second hand, it already had one,
but I didn’t know what it was.
In order to give the card an IP, you have 2 obvious options

1.  serial cable and terminal emulator

2.  Ethernet and ARP

As I didn’t have the “special” serial cable,
I decided to go the Ethernet route.
I would need the MAC address.
The subnet mask and default gateway also need to be set up.
Pg 11 of APC_ap9606_installation_guide.pdf goes through the procedure.
All APC devices have a MAC address that begin with 00 C0 B7
Although my network management card had a sticker with the MAC address on it.
“You may want to check your DHCP client list for any MAC addresses beginning with 00 C0 B7,
which indicates an APC address.
In addition, check the card you are trying to configure.
Any card with valid IP settings will have a solid green status LED”.

When I received my AP9606 Web SNMP Management Card, I didn’t have a clue what the IP address had been set to.
If it was a new card it wouldn’t have yet been set and I would be able to easily set it without having to workout
what its subnet was.
On Pg 11 of the “Web/SNMP Management Card Installation Manual”
It goes through setting up an IP from scratch using ARP.
So I plugged my notebook into the AP9606’s Ethernet port and spun up Wireshark.

What you’ll generally be looking for is a record with the Source looking like

"American_[last 3 bytes of MAC]"

Time                    Source                Destination    Protocol    Info
231    715.948894    American_42:6f:b1    Broadcast    ARP        Who has 10.1.80.3?  Tell 10.1.80.222

And an ARP request that looks something like the following…
The first 3 bytes of the MAC will always be 00-C0-B7 for a AP9606.

Address Resolution Protocol (request)
 Sender MAC address: American_42:6f:b1 (00:c0:b7:[3 more octets here])
 Sender IP address: 10.1.80.222 (10.1.80.222)
 Target MAC address: American_42:6f:b1 (00:c0:b7:[3 more octets here])
 Target IP address: 10.1.80.3 (10.1.80.3)

I set the notebook to use a static IP of
10.1.80.2/24
and default gateway to the Target IP of
10.1.30.3
You may have to play around a bit with the subnet mask until you get it right.
I was just lucky.
Then tried using ARP to assign the new IP address,
but it wasn’t sticking.
So I tried to telnet in and was prompted for a username and password.
The default of apc for both was incorrect so obviously it had already been altered.
There is also another account of u- User p- apc
but this didn’t exist or had been changed.
So I contacted APC for the backdoor account as discussed here
and was directed to here.
This is no good unless you have a special serial cable which I didn’t.
I asked for the pin layout of the cable and was told,
that they make them,
but don’t know what the pin layout is.
The nice fellow at APC support directed me to a cable to buy.
A little pricey at $100NZ,
for a single use cable.
There is no proper way to reset the password by the Ethernet interface.
This left me with two obvious options.

1.  Make up a serial cable with I believe…

Pin#2 Female to Pin#2 Male,
Pin#3 Female to Pin#1 Male,
Pin#5 Female to Pin#9 Male,
and find a computer with a com port.
Layout info found here
It was correct.
That would cost next to nothing.

2.  just crack the credentials with one of these.

The second seemed like it would be the path of least resistance immediately (this turned out to be incorrect),
as I had the software, but not enough parts for a serial cable.
THC-Hydra seemed like a good option.
Once I downloaded and ran Hydra, I received the following error

 5 [main] ? (1988) C:\cygwin\bin\bash.exe: *** fatal error - system shared
 memory version mismatch detected - 0x75BE0074/0x75BE0096.
 This problem is probably due to using incompatible versions of the cygwin DLL.
 Search for cygwin1.dll using the Windows Start->Find/Search facility
 and delete all but the most recent version.  The most recent version *should*
 reside in x:\cygwin\bin, where 'x' is the drive on which you have
 installed the cygwin distribution.  Rebooting is also suggested if you
 are unable to find another cygwin DLL

This error is due to having incompatible versions of cygwin1.dll on your system.
So did a search for them and found that my SSH install had an older version of cygwin1.dll.
So renamed it,
and still had problems,
rebooted, and all was good.
the only cygwin1.dll should be in the same directory that hydra.exe is run from.

How to use THC-Hydra

Some good references here…
http://www.youtube.com/watch?v=kzJFPduiIsI
http://www.pauldotcom.com/2007/03/01/password_cracking_with_thchydr.html

Command I used.

C:\hydra-5.4-win>hydra -L logins.txt -P passwords.txt -e n -e s -o hydraoutput.txt -v 10.1.80.222 telnet "Welcome hacker"

I got a false positive of User name n/a Password steven
So rather than spend more time on populating the logins.txt and passwords.txt.

I decided to try the serial cable route

As it turned out, I wouldn’t have guessed the username,
found this out once I logged on using the serial interface.
This is the pinout I used.

This is the single use cable I made.
Total cost of $0.00


Make sure you’re all plugged in.
I used minicom as my terminal emulator to connect to the UPS’s com port.
Installation and usage details here.

You need to make sure you’re serial port/s are on in the BIOS.
I didn’t check mine, but they were on.

Need to make sure Linux knows about your serial port/s
Run the following command:

Use setserial to provide the configuration information associated with your serial ports.

Configuring your serial ports.

To setup your terminal emulator (minicom in my case):

$ minicom -s -c on

Choose “Serial port setup”
and you will be presented with a menu like the following.

This is where you get to set the following:
2400 BPS, 8 databits, No parity,
one stop bit and flow control is set to none.
Then select Save setup as dfl

Exit.

You should now be prompted for authentication from the Smart-Ups.

Or you can choose “Exit from Minicom” and run

$ minicom -c on

later.
If you get output like…

Device /dev/ttyS[number of your port here] is locked.

You’ll have to

# rm /var/tmp/LOCK..ttyS[number of your port here]

Now is where you get to log on as the default user/pass apc/apc
Press the reset button on the AP9606
and press Enter key,
then repeatedly if necessary.
This is poking the AP9606 in order to get a login prompt
Once you get the User Name,
you can enter the “apc” user (without the quotes) and then for the Password,
“apc” (without the quotes).
You have a 30 second window here to login.
Else you have to repeat the reset process and try again.

From the Control Console menu,
select System, then User Manager.
Select Administrator,
and change the User Name and Password settings,
both of which are currently apc.

I also changed the IP settings.
From the Control Console,
select
2- Network
1- TCP/IP
and change your IP settings.

There are quite a few settings you can change on the card,
you should just be able to follow your nose from here.
You’ll also want to make sure the Web Access is Enabled.
Take note of the port also, usually 8000.
Changing the password via the serial interface is also detailed here.
This post was also quite helpful.

Changed the IP settings back to how they were on my notebook.
Could now connect via telnet and HTTP.
Turned md5 on to try and boost the security of passing credentials to the web UI.
Turned out the jre is also needed for this.
Went through that process and it was looking promising,
but the web UI no longer accepted my password.
Not sure why this is,
but it means if you want to be secure when you log into the web UI,
you are going to have to plug your Ethernet cable directly into the AP9606.
Otherwise your passing credentials in plan text.

Upgrade of firmware

The latest firmware is found here.
Directions on upgrading are found here.
In saying that, APC recommended I use the earlier aos325.bin and sumx326.bin from here if using Windows XP.
Some details around the firmware required for the different management card types for use in a Smart Slot equipped APC UPS
The firmware version is found under Help->About System on the NMC’s Web interface.

Using PSCredentials

June 2, 2011

I’ve been working on a small project that shuts down machines attached by network and of course power feed to an APC Smart-UPS.
The code that was shutting down the guests required authentication to be passed to the receiving services.

I decided to give the following PowerShell cmdlets a try.

  • Get-Credential
  • ConvertTo-SecureString

———————————————————————————-

Script that creates the password file

(Set-Credential.ps1) looks like this:

Param($file)
$credential = Get-Credential
$credential.Password | ConvertFrom-SecureString | Set-Content $file

Get-Credential prompts for a username and password and creates the PSCredential associating the password with the username.
ConvertFrom-SecureString from the PS documentation…
The ConvertFrom-SecureString cmdlet converts a secure string
(System.Security.SecureString) into an encrypted standard string (System.String).
Then writes the string to the file specified.

Set-Credential can be invoked like this:

C:\Scripts\UPS\Set-myCredential.ps1 C:\Scripts\UPS\mp.txt

———————————————————————————-

Script that reads the password file

(Get-Credential.ps1) into a SecureString.
Then creates the PSCredential based on the username provided and the password as a SecureString.
Then returns the PSCredential:

Param($user,$passwordFile)
$password = Get-Content $passwordFile | ConvertTo-SecureString
$credential = New-Object System.Management.Automation.PsCredential($user,$password)
$credential

———————————————————————————-

By the look of it, when creating the encrypted password Get-Credential adds some machine specific information.
As the password file is not machine agnostic (can’t be shared or tranfered).

From my PowerShell script that loaded the assembly into memory and started the shutdown procedure, it looked something like this.

param (
   [parameter(Mandatory=$true, position=0)][string] $scriptPath,
   [parameter(Mandatory=$true, position=1)][string] $fileServerName,
   [parameter(Mandatory=$true, position=2)][string] $fileServerUser,
   [parameter(Mandatory=$true, position=3)][string] $vSphereServerName,
   [parameter(Mandatory=$true, position=4)][string] $vSphereServerUser
)

Set-StrictMode -Version 2.0
# Creates a .net assembly in memory containing the PowerOffUPSGuests class.
# Then we call the InitShutdown passing the details of the machines that need to be shutdown.

$credentialRetrievalScript = Join-Path -Path $scriptPath -ChildPath 'Get-Credential.ps1'
$fileServerUserPwLocation = Join-Path -Path $scriptPath -ChildPath 'FileServerPw.txt'
$vSphereServerUserPwLocation = Join-Path -Path $scriptPath -ChildPath 'VMHostPw.txt'

# names of the ServerController's I.E. the collection of servers that will be shutdown
# these class's need to exist in the $scriptPath and derive from ServerController
$freeNASController = 'FreeNASController'
$vMServerController = 'VMServerController'

# instantiate the credential objects
$fileServerCredential = & $credentialRetrievalScript $fileServerUser $fileServerUserPwLocation
$vSphereServerCredential = & $credentialRetrievalScript $vSphereServerUser $vSphereServerUserPwLocation

# add the assembly that does the work.
Add-Type -Path .\PowerOffUPSGuests.dll

# instantiate a ServerAdminDetails for each server we want to shutdown
$fileServerAdminDetailsInstance = New-Object -TypeName BinaryMist.Networking.Infrastructure.ServerAdminDetails -ArgumentList $freeNASController, $fileServerName, $fileServerCredential
$vSphereServerAdminDetailsInstance = New-Object -TypeName BinaryMist.Networking.Infrastructure.ServerAdminDetails -ArgumentList $vMServerController, $vSphereServerName, $vSphereServerCredential

# instantiate a PowerOffUPSGuests
$powerOffUPSGuestsInstance = New-Object -TypeName BinaryMist.Networking.Infrastructure.PowerOffUPSGuests

# create generic queue and populate with each of the ServerAdminDetail items
# ServerAdminDetails is the base class of FileServerAdminDetails and vSphereServerAdminDetails
$serverAdminDetailsQueueInstance = .\New-GenericObject System.Collections.Generic.Queue BinaryMist.Networking.Infrastructure.ServerAdminDetails
$serverAdminDetailsQueueInstance.Enqueue($fileServerAdminDetailsInstance)
$serverAdminDetailsQueueInstance.Enqueue($vSphereServerAdminDetailsInstance)

$powerOffUPSGuestsInstance.InitShutdownOfServers($serverAdminDetailsQueueInstance)

To debug my library code, I needed to run it somehow.
So I just wrote a small test which passed the PSCredential instance to the code that was going to shutdown the UPS guest.

private PSCredential GetMyCredential(string userName, string pWFileName) {

   string encryptedPw;
   using (StreamReader sR = new StreamReader(pWFileName)) {
   //read the encrypted bytes into a string
   encryptedPw = sR.ReadLine();
   }

   PSCredential pSCredential;
   using(SecureString pW = new SecureString()) {
      char[] pWChars = encryptedPw.ToCharArray();
      foreach(char pWChar in pWChars) {
         pW.AppendChar(pWChar);
      }
      pSCredential = new PSCredential(userName, pW);
   }
   return pSCredential;
}

[TestMethod]
public void TestInitFileServerShutdown() {
   _powerOffUPSGuests = new PowerOffUPSGuests(ConfigurationManager.AppSettings[LogFilePath]);

   PSCredential fileServerCredential = GetMyCredential(
      ConfigurationManager.AppSettings[FileServerUser],
      Path.GetFullPath(ConfigurationManager.AppSettings[FileServerUserPwFile])
   );

   _powerOffUPSGuests.InitFileServerShutdown(ConfigurationManager.AppSettings[FileServer], fileServerCredential);
}

Inspiration