Archive for the ‘Projects’ Category

350Z Hi-fi Install

February 28, 2015

When we acquired the 2004 350Z, it had a low quality hi-fi installed in it. The FM radio broadcast band in Japan is 76-90 MHz Which misses a large portion of the New Zealand stations.

It’s been a few years since I performed a car hi-fi installation, so I lent on some friends. Specifically Matthew Fung which sold me the AVK6’s and the LRx 5.1MT.

Previous Hi-fi install

Now this install was in my VY Ute. This image shows the sub woofer enclosure in place under the deck right next to the fuel tank. The metal deck panel gets fixed over this and then the tray liner goes over top of that.

VY sub woofer enclosure in place

 

Unlike the 350Z install, this enclosure didn’t have to be pretty at all because it would never be seen.

VY sub woofer and enclosure

 

I learnt a lot from this install that I would take forward to future installs. Namely the 350Z install

350Z Hi-fi Install

Some of Matthew’s advice was to spend in the following fashion:

35% of your budget on speakers, 35% on amplifier, 20% head unit and 10% on sub woofer. This is good advice.

Required Components

Other than the 350Z

  • Head unit: Alpine CDE-148EBT From Quality Car Audio
  • DIN/DDIN kit: 99-7402 from Quality Car Audio
  • Power amplifier: Italian made Audison LRx 5.1MT from Matthew Fung
  • Front speakers: Audison AVK6 6.5’s from Matthew Fung
  • Rear speakers: Existing factory from Matthew Fung
  • Sub woofers: 2 x Alpine SWR-12D4 From Quality Car Audio. One of the reasons for two sub woofers was appearance.
  • Sub woofer box: Fabricated with some old 19mm MDF cover sheets that have been in my garage since I left the carpentry trade about 15 years ago.
  • Square drive (because they are far superior to posi) super screws
  • Sound deadening: Single layer of Soundstream Deathmat from HyperDrive. Once finished about 15kg will be applied to floors, both front and boot and front doors.
  • In-line fuse block: 80amp from JayCar
  • Battery terminals: from JayCar
  • Power cable: 8m of STINGER SPW14TR 4 AWG GAUGE from Ebay
  • Speaker cable: Stinger SHW512B 100 ft Roll of HPM 12 Gauge Matte Blue Car Speaker Wire run to all speakers from Ebay
  • RCA cables: 3 x Stinger SI8217 Audio RCA Interconnect Cable 2 Channels 8000 Series 17 ft from Ebay
  • Internet of course
  • Carbon fiber vinyl wrap
  • Front speaker mounts: Rubber drain pipe transitions
  • Approximately one week labour

Yes I was tempted to skimp on the likes of the power cable, speaker wire, RCA cables because the good ones are much dearer than the cheap ones. This is where you really get what you pay for. If you want to produce a great result, aim for the best you can get here. A few hundred dollars really makes a big difference. I’ve used cheaper parts here on previous installs and the overall difference is very noticeable. It also give you room to upgrade components in the future without having to run all your wires again which is where most of the install time gets sucked up.

Day One

Design and Construct Sub Woofer Enclosure

Generally speaking if you are running two of the exact same sub woofers in the same air space you can just combine the air spaces, but if you swap one of the sub woofers for one that behaves differently, then you are likely to run into issues where the sound waves interfere with each other. Also if you have a sub woofer enclosure that has an air space for each sub woofer you also get bracing for free from the divider. Bracing is all about making the enclosure more rigid so that the sound waves don’t cause the casing to vibrate/move with the waves more than it should. The more rigid you can make the enclosure the more you help the enclosure do what it’s designed to do… produce accurate low frequencies.

sub woofer enclosure

I chose to have one enclosure with two separate air spaces, thus providing a rigid casing. There are three distinct volume calculations here.

  1. One for the main air space
  2. One for the arch way (Looks like a ‘D’ rotated 90° counter clockwise) you can see cut in the rear of the enclosure
  3. The space behind the rear of the main enclosure. This had to be done to accommodate the cast aluminium frame of the SWR-12D4.

I also allowed for adjustment in air space with the two ends of the enclosure, as they could be moved in or out depending on whether I needed more or less air space. As you can see in the image above, the rear and the front of the enclosure have tapered cuts down toward the bottom of the enclosure. These tapered cuts were made once A) the box had been constructed, B) the ends had been adjusted and fixed in place and C) I was certain of the internal air space.

All joints are wood glued and sealed with an acrylic sealant. Solvent based sealants can play havoc with some of the sub woofer internals, thus causing premature failure. The screws are varying lengths depending on the angle of the join. All screw holes were pre-drilled into the initial MDF that needs to be retained with the diameter of the thread, so that the thread doesn’t grip onto the MDF being retained. Then the MDF that the screws actually bite into are pre-drilled with a bit the diameter of the shank (which is a little smaller than the thread). If you don’t do this second part, the MDF will just split which means very little grip and essentially a very weak joint.

All screws were countersunk with a… countersink bit. This is done so that when you cover the box in your chosen covering, it all appears flat. Two circular holes cut with jig saw. The holes that the speaker cable passes through are also sealed with acrylic sealant.

I also chose not to use terminals on the sub woofer enclosure to fix speaker wires to, but rather run the speaker wires straight from the power amplifier to the sub woofer, thus removing one of the couplings. The sub woofer enclosure can still be disconnected at the power amplifier and at the speaker. If you’re planning on swapping sub woofers in and out frequently,

As I’m a carpenter by trade (from aprx 1990 – 2000), this construction was trivial.

For those of you unfamiliar with the 350Z, this is what the target space looks like (plan view) with the dimensions:

350Z sub woofer space

 

You’ll also notice the cross section arrows. They should actually be facing the other direction. The A:A section is the below image in reverse. If I was to go full width with the enclosure, it wouldn’t be able to be installed as the rear hatch opening is considerably narrower than the enclosures target location. It ends up being just over 100mm gap on each side of the enclosure once installed. I just fill this space with plumbers foam pipe lagging which actually looks fine. There is also a small gap in front of the enclosure (between enclosure and rear speakers) and behind it (between enclosure and strut brace). I’ve just pushed 12mm black PEF rod down the gap and it again looks fine. One of the rules in carpentry is, if you can’t hide a transition, then emphasis it. That’s what we’ve done with these gaps. The enclosure doesn’t need to be fixed in place in case of head on collision, as it can’t move forward but can tilt forward a little, but as the roof is too low to allow it to move forward far, it’s safe. Just make sure those huge sub woofers are well fixed as they would be like canon balls in an accident.

Volume Calculations

The gross internal target volume for the sub woofer was 0.85 ft³. It’s range is 0.23 ft³ both ways. I ended up being just below 0.85 ft³. So I was happy.

  1. For the main air space, that’s the largest one. I just worked out the area of the quadrilateral (bottom, front, top, rear) then multiply by the depth. All in cm’s, then convert cm³ to ft³. As I’m lazy, the easiest way is to just throw your values into the calculator here or here. These dimensions are all internal and specified in mm unless stated otherwise.
    EnclosureQuadrilateral
  2. For the semi-circle (‘D’ rotated 90° counter clockwise) it’s just πr2/2 for the area then multiplied by the depth of 1.9cm
  3. The following was the calculation for the triangle behind the ‘D’. this is also used to mount the power amplifier. The outer panel here carries across two of these internal spaces. Hope that makes sense? You’ll see an image below anyway of how the power amplifier is mounted on the rear of the enclosure. I also had the very rear panel of the enclosure over-hang the internal dimension by about 50mm. This was so that I could mount some LED strip lighting if I wanted to. The Audison LRx 5.1MT has it’s logo light up which has for now satisfied my geek addiction for LED mood lighting.
    enclosure triangle volume calculation

 

SWR-12D4 Dimensions

SWR-12D4 dimensions

SWR-12D4 Design

SWR-12D4 design

 

Day Two

Just more of the same thing plus fitment and making small adjustments. I found that by draping a towel over the rear strut brace and just sliding the enclosure over it and tilting it forward it’d just slot into place nicely.

Day Three

Removal of Internals

Removed all the internal panels that I’d need in order to run cables and apply sound deadening. I numbered each piece I pulled off in order, so that it was easier to work out the re-fit order when the time came. I also mask any plugs or screws etc to the panel so they don’t get lost.

  • Remove center console
  • Remove panels to get to rear speakers and to run power cable and speaker wires. I also decided to make use of the empty space which is about 2 ft³ behind the drivers seat. For now this is still left open. I’m looking around wreckers for another cubby-hole like the one behind the passenger seat, as they are identical. This will allow me to claim back the space that we’ve lost due to the enclosure going into the limited boot space.
  • Removed door sill panels and kick panels. Both of which can be pulled off gently.
  • Remove seats. Also followed the sequence in this post to reset the windows going all the way up.

Once I was done with the full install and started the car, the Engine Management Light (EML) came on. This will continue to happen until you reset it. First thing to do though is find out what the fault is if there is one. Follow the sequence of events here. Diagnose the fault code based on the flashes detailed here and here. My code ended up being “P0000” (No Self Diagnostic Failure Indicated). That’s:

10 (0) Long Blink
10 (0) Short Blinks
10 (0) Short Blinks
10 (0) Short Blinks

So I reseted the ECU and I was done.

Apply Sound Deadening

The 350Z’s have a lot of road noise by default. If you want to keep that out so that it doesn’t interfere with your new sounds and keep your sounds inside the car it can be a good idea to apply this liberally. Plus it’s a good time to do this when you have most of your internal panels removed. I’ve found that it made quite a big difference. The sounds are barely audible outside the vehicle even when played at a modest level. When cranked, you can still feel the low frequencies outside more than you can hear them. Believe it or not, that includes the lower frequencies. I applied this a little higher than you can see in the below image. All the rattles you hear in a lot of vehicles with large sub woofer setups are gone. This means the low frequencies are not loosing energy in the panels but are instead going into the cabin space. Which is exactly where we want them.

Sound Deadening to floors

Day Four

Power from Battery to Power Amplifier

Run power cable from battery through firewall to boot of car where amplifier will sit. This link shows a left hand drive 350Z. For right hand drive Z’s, it’s pretty similar.

Run RCA Leads

from front console to where sub woofer enclosure will live. Careful to keep speaker and RCA leads as far away from power cables as possible to reduce interference.

Here you can see the power cable on the left of the drive-shaft tunnel (top of image) and the three RCA leads on the right of the drive-shaft tunnel (bottom of image) zip-tied together and passing through the body just behind where the carpet finishes. You can also see the RCA lead ends looped around the gear stick in the image waiting to be plugged into the new head unit once installed.

Also notice the fold-out work light far left of the image. This is cordless and has magnets on the rear of both LED panels and the middle shaft. The LED panels can be pivoted around the shaft. This is one of the most handy work lights I’ve ever used. It also has a torch light in the middle shaft. This can be run with A) one panel running, B) two panels running, C) just the torch light running. It’s rechargeable and seems to run for as long as I need it each day. I think it’s supposed to run for three hours with only one panel running, but it seemed to run for longer than that. $30 from bunnings.

RCA leads and power cable

Run Speaker Wires

All except front left door and sub woofers.

For the front speakers, the hardest part of just about the entire install is getting the 12 gauge speaker wire through the plastic door harness without joiners. This post provided the detail I needed. Previous car hi-fi installs have been similar in that this task has been the most frustrating. I started with the right door (drivers door) which is harder than the passenger door for several reasons.

  • There is next to no visibility from underneath the steering wheel looking at where the speaker wire must pass through the plastic harness
  • The plastic harness has a greater population of used pins that you have to be very careful not to hit with your knife or drill. I used both tools.

The other thing to keep in mind is that the plastic harness that clicks into place on the car body must go in top first from the inside, not bottom first. This cost me a bit of time (I bent the metal retainer out of shape because I put bottom in first) and I ended up coming back to the driver side after I’d done the passenger side successfully and learnt a few more tricks.

For the front speakers, I ran the speaker wires down the door sills right next to the plastic clips that hold the sill covers on. Then through the holes at the back of the sill and through into the boot near where the power amplifier would be mounted.

I also ran 12 gauge wire for the rear speakers. There are plenty of holes to thread them through and keep them well concealed. Keeping in mind that they need to stay as far away from power cables as possible.

 

Factory Speaker Wire Colours

I took note of factory speaker wire colours in case I needed them later. I didn’t, but here they are anyway for anyone needing them:

Front Left

  • Red with silver loops
  • Blue with silver loops

Front Right: Didn’t capture these

Rear Left

  • Light green with silver loops
  • Light brown with dark brown stripes

Rear Right

  • Light orange with silver loops
  • Black and pink stripes with silver loops

 

Day Five

Fit Tweeters

See the image on Day Six for how the tweeter looks mounted. The small plastic triangle panel that the tweeter is fixed to from memory needs to be pulled in toward the car where it meets the window pane and then slid down. Mine had an existing round grill that looked like a tweeter was mounted underneath, but all it was was a grill. I drilled a hole just big enough to pass the tweeters existing wire through it. At this stage I’ve used double sided sponge tape. The tweeters are fairly heavy so we’ll wait and see if the tape continues to hold them on. If it doesn’t I’ll have to resort to some glue. Currently it’s been about a week and they’re still staying put. I used some acetone to clean the surface of both the rear of the tweeter and the area that it was going to be fixed to. Just be careful with this though, as it will start to melt the plastic, so make sure you only do it where it’s not going to be seen.

As you can see in the image, no wires are visible and it’s a good position for the sound stage. I’ve got the crossovers set to the highest gain for the tweeters. I thought this may have been to much, but it seems perfect. The lower frequencies are surprisingly well handled by the Audison AVK6 6.5″ drivers.

Run Speaker Wire to Left Front Door

Similar to the right door.

 

Day Six

Apply Sound Deadening to Side Doors

A lot of road noise comes in the doors on the 350Z’s. Also with your hi-fi setup, you’d loose a lot of energy through your doors. Applying sound deadening liberally to the doors as well as the floors stops a lot of this. It’ll also kill any rattles you may have had otherwise.

Fit Front Speakers with Crossovers

There is a sunken space where I fixed my crossovers. I soldered and crimped terminals to the speaker wires that get screwed to the crossover. I soldered and applied heat-shrink sleeving to the tweeter wires / 12 gauge wire junction.

Audison AVK6 6.5

 

Below is a closer look at what the 6.5″ and tweeter mounts look like. I used a drainage transition rubber pip for the 6.5″ driver. These are great for mitigating vibration and for cutting to the exact right size.
Now you have to get the off-set right here. The speaker from rear of rim to back is 70mm. I cut my rubber mounts to 40mm. This allowed 30mm sitting inside the door panel. The window pane when it’s wound down goes behind the speaker, so you don’t have a lot of room there. I think you’ve got about 35mm, so I had about 5mm clearance. If it’s not enough, the glass is going to hit the rear of your speaker and could be disastrous. 40mm mounts were actually to wide. It was quite hard to fit the door panels on as they were touching the speaker rim. I’d suggest making your mounts anything over 35mm (don’t go less), target should be about 36/37mm. So as you can see, this has got to be fairly accurate else either your window will hit the rear of the speaker or your door panel will be up against the speaker rim.
There is an excellent step by step guide to most of this here.

TweeterAndDoorSpeaker

Fit Rear Speakers

I was using the existing factory speakers as they don’t really matter that much. Why don’t they matter much? Most of your sound stage should be coming from the front of you.

In my case there were no signs of which speaker terminal was positive or negative. The best way to work this out is to use a 9v battery and touch wires from the batteries + and – terminals to each of the speakers terminals. if the speaker pops out slightly when connected, it means you’ve connected the batteries + to the speakers +. If the speaker pops in slightly when connected, it means you’ve connected the batteries + to the speakers -. This way you can tell which terminal you should connect your designated positive speaker wire. This method doesn’t harm your speaker as the voltage is too low.

 

Day Seven

Install Head Unit

Most of the steps you would need are here.

Fit DIN/DDIN kit 99-7402 to the dash assembly. As the CDE-148EBT is a single DIN unit, we get some space back in the form of another compartment below the head unit.

The Alpine CDE-148EBT comes with a harness. I only needed to solder -> heat shrink on the following harness wires to the existing wires. The existing wires were all factory labelled with tags. This was obviously very helpful. I had already located a wiring guide for this operation that I would not now need. Strange thing was with the guide that the colours were incorrect for my Z.

  • Orange (head unit illumination)
  • Red (ignition)
  • Yellow (battery)
  • Black (ground) just replaced existing ground wire
  • Blue/white (remote turn on, supplies 12v to power amp to tell it to turn on when head unit is on) I missed this one and had to pull the console out again and connect it. I go through this in day eight.

Connect Aux in, USB extension lead, all six RCA’s.

There are a lot of wires behind the head unit and it can be a bit tricky getting everything back in. Be gentle and patient.

Re-fit Seats and Some Panels

Self explanatory.

Apply Sound Deadening

Around where sub woofer enclosure is going to live. I applied it to the sides also.

SoundDeadeningAroundEnclosure

Fit 80 amp In-line Fuse

I fitted the fuse block about 150mm from the battery. The end of the in-line fuse block is just visible in the below image.

80 amp inline fuse

Install Sub Woofer Box

EnclosureFromTop

Fit Sub Woofers

If you plan on swapping your subs in and out a bit then I’d advise using the likes of these inserts to screw into. Screw these into your MDF then screw into them:

EnclosureMDFScrewFixings

It was about here that I realised that I should have gone with the  SWR-12D2 (the 2 ohm version) sub woofers. As I hadn’t actually used the sub woofers yet, I’d be able to just exchange the 4 ohm subs with the 2 ohm subs. Quality Car Audio refused to provide an exchange for the still new subs. Both 4 ohm and 2 ohm models where exactly the same price.

Some reading on under-powering sub woofers:

Connected Power Cable to Amplifier

Ran Earth Cable to Bare Metal

Unscrewed a bracket that came through the left side of the car in the spare wheel well and fitted the terminal I had soldered and crimped onto my 4 gauge earth cable between the bracket and the body of the car after I sanded the paint off for good connection. Cranked that bolt up nice and tight.

Connected all Speaker Wires to Power Amplifier

Connected RCA Leads

Test Front and Rear Speakers

Re-install internal panels

 

Day Eight

Power Amplifier Not Auto Powering On

Take power amp in to be tested as it wouldn’t power on. The car power amplifiers I’d used in the past would turn on when they receive signal from the pre-outs. That’s signal that’s pre-amplified. Turns out a jumper, well a plastic jumper with a single 12v remote turn on from the head unit was needed on the bottom left pin of the power amplifier. So out with the head unit again and run the extra wire. That’s why you want to leave re-fitting of panels as late as possible.

Cleaned up

Charged Battery

Initial Tuning

Power Amplifier

Currently I have the Audison LRx 5.1MT gains set to (where 0 is 0 and 10 is max):

  • Fronts: 8
  • Rears: 3 (existing very crappy factory speakers)
  • mono Sub woofer channel: 8

Head unit

  • Turn off internal power amp as it’s not used and helps to reduce power supply interference/noise
  • The Alpine CDE-148EBT has a lot of options and features when it comes to tuning your sound. This will keep me busy for a long time I’d say.

Sub Woofer Configuration

  1. I first tried 2 ohm setup with one speaker as a benchmark because I knew that it was ideal
  2. I then tried parallel 1 ohm which I don’t think provided enough energy to the sub woofers
  3. Third configuration I tried was in series 4 ohm which worked quite well.

I’ve also noticed that with the sub woofers pointing forward so I can actually hear them may not have been the best design decision. Especially with two of them running in mono as it affects the sound stage a bit. it probably would have produced a slightly better result if they had of been pointing toward the rear of the car under the strut brace so that the bass could be felt more than heard. In saying that, it’s still early days and I have a lot of tuning to get the levels just right. Also as I didn’t want to remove the spare wheel which sits under the boot base in order to mount the power amplifier under there, one of the only places left would be to mount it on top of the sub woofer enclosure, which although it is rather a good looking piece of hardware. I think 12″ sub woofers look better.

EnclosureFromFront

A bit more reading on parallel verses series wiring of your sub woofers.

Mounted Power Amplifier to Rear of Sub Woofer Enclosure

Audison LRx 5.1MT

Overall Sound

I’m kind of surprised that I’ve managed to achieve such a truthful sound of my recordings. I wasn’t sure this was possible in a car. Think of studio monitors and that’s what this system reminds me of. All of a sudden many recordings sound bad and the good recordings sound outstanding and produce the emotion that high quality music (recording, mix, mastering) is renowned for. For example my personal recordings that I did a few years ago sound amazing as I had the help from one of New Zealand’s best sound engineers / producers (Ian McAllister).

I think also the fact that I didn’t take any short-cuts that I’m aware of? It’s all the small details that add up as well as using great components.

Ongoing

Hunting down boot carpet for reduced space. The factory carpet in the Z is not actually carpet but just felt. So I’m going to get some real carpet with the ‘Z’ embroidered into it. There are a few places that actually supply this.

 

Installation and Hardening of Debian Web Server

December 27, 2014

These are the steps I took to set-up and harden a Debian web server before being placed into a DMZ and undergoing additional hardening before opening the port from the WWW to it. Most of the steps below are fairly simple to do, and in doing so, remove a good portion of the low hanging fruit for nasty entities wanting to gain a foot-hold on your server->network.

Install and Set-up

Debian wheezy, currently stable (supported by the Debian security team for a year or so).

Creating ESXi 5.1 guest

First thing to do is to setup a virtual switch for the host under the Configuration tab. Now I had several quad port Gbit Ethernet adapters in this server. So I created a virtual switch and assigned a physical adapter to it. Now when you create your VM, you choose the VM Network assigned to the virtual switch you created. Provision your disks. Check the “Edit the virtual machine settings before completion” and Continue. You will now be able to modify your settings before you boot the VM. I chose 512MB of RAM at this stage which is far more than it actually needs. While I’m provisioning and hardening the Debian guest, I have the new virtual switch connected to the clients LAN.

ESX Network Configuration

Once we’re done, we can connect the virtual switch up to the new DMZ physical switch or strait into the router. Upload the debian .iso that you downloaded to the ESXi datastore. Then edit the VM settings and select the CD/DVD drive. Select the “Datastore ISO File” option and browse to the .iso file and select the “Connect at power on” option.

6_NewVMSelectIso

Kick the VM in the guts and flick to the VM’s Console tab.

OS Installation

Partitioning

Deleted all the current partitions and added the following. / was added to the start and the rest to the end, in the following order.
/, /var, /tmp, /opt, /usr, /home, swap.

Partitioning Disks

Now the sizes should be setup according to your needs. If you have plenty of RAM, make your swap small, if you have minimal RAM (barely (if) sufficient), you could double the RAM size for your swap. It’s usually a good idea to think about what mount options you want to use for your specific directories. This may shape how you setup your partitions. For example, you may want to have options nosuid,noexec on /var but you can’t because there are shell scripts in /var/lib/dpkg/info so you could setup four partitions. /var without nosuid,noexec and /var/tmp, /var/log, /var/account with nosuid,noexec. Look ahead to the Mounting of Partitions section for more info on this.
In saying this, you don’t need to partition as finely grained as you want options for. You can still mount directories on directories and alter the options at that point. This can be done in the /etc/fstab file and also ad-hoc (using the mount command) if you want to test options out.

You can think about changing /opt (static data) to mount read-only in the future as another security measure.

Continuing with the Install

When you’re asked for a mirror to pull packages from, if you have an apt-cacher[-ng] proxy somewhere on your network, this is the chance to make it work for you thus speeding up your updates and saving internet bandwidth. Enter the IP address and port and leave the rest as default. From the Software selection screen, select “Standard system utilities” and “SSH server”.

10_SoftwareSelection

When prompted to boot into your new system, we need to remove our installation media from the VMs settings. Under the Device Status settings for your VM (if you’re using ESXi), Uncheck “Connected” and “Connect at power on”. Make sure no other boot media are connected at power on. Now first thing we do is SSH into our new VM because it’s a right pain working through the VM hosts console. When you first try to SSH to it you’ll be shown the ECDSA key fingerprint to confirm that the machine you think you are SSHing to is in fact the machine you want to SSH to. Follow the directions here but change that command line slightly to the following:

ssh-keygen -lf ssh_host_ecdsa_key.pub

This will print the keys fingerprint from the actual machine. Compare that with what you were given from your remote machine. Make sure they match and accept and you should be in. Now I use terminator so I have a lovely CLI experience. Of course you can take things much further with Screen or Tmux if/when you have the need.

Next I tell apt about the apt-proxy-ng I want it to use to pull it’s packages from. This will have to be changed once the server is plugged into the DMZ. Create the file /etc/apt/apt.conf if it doesn’t already exist and add the following line:

Acquire::http::Proxy "http://[IP address of the machine hosting your apt cache]:[port that the cacher is listening on]";

Replace the apt proxy references in /etc/apt/sources.list with the internet mirror you want to use, so we contain all the proxy related config in one line in one file. This will allow the requests to be proxied and packages cached via the apt cache on your network when requests are made to the mirror of your choosing.

Update the list of packages then upgrade them with the following command line. If your using sudo, you’ll need to add that to each command:

apt-get update && apt-get upgrade # only run apt-get upgrade if apt-get update is successful (exits with a status of 0)


The steps you take to harden a server that will have many user accounts will be considerably different to this. Many of the steps I’ve gone through here will be insufficient for a server with many users.
The hardening process is not a one time procedure. It ends when you decommission the server. Be prepared to stay on top of your defenses. It’s much harder to defend against attacks than it is to exploit a vulnerability.

Passwords

After a quick look at this, I can in fact verify that we are shadowing our passwords out of the box. It may be worth looking at and modifying /etc/shadow . Consider changing the “maximum password age” and “password warning period”. Consult the man page for shadow for full details. Check that you’re happy with which encryption algorithms are currently being used. The files you’ll need to look at are: /etc/shadow and /etc/pam.d/common-password . The man pages you’ll probably need to read in conjunction with each other are the following:

  • shadow
  • pam.d
  • crypt 3
  • pam_unix

Out of the box crypt supports MD5, SHA-256, SHA-512 with a bit more work for blowfish via bcrypt. The default of SHA-512 enables salted passwords. How can you tell which algorithm you’re using, salt size etc? the crypt 3 man page explains it all.
So by default we’re using SHA-512 which is better than MD5 and the smaller SHA-256.

Now by default I didn’t have a “rounds” option in my /etc/pan.d/common-password module-arguments. Having a large iteration count (number of times the encryption algorithm is run (key stretching)) and an attacker not knowing what that number is, will slow down an attack. I’d suggest adding this and re creating your passwords. As your normal user run:

passwd

providing your existing password then your new one twice. You should now be able to see your password in the /etc/shadow file with the added rounds parameter

$6$rounds=[chosen number of rounds specified in /etc/pam.d/common-password]$[8 character salt]$0LxBZfnuDue7.n5<rest of string>

Check /var/log/auth.log
Reboot and check you can still log in as your normal user. If all good. Do the same with the root account.

Using bcrypt with slowpoke blowfish is a much slower algorithm, so it’s even better for password encryption, but more work to setup at this stage.

Some References

Consider setting a password for GRUB, especially if your server is directly on physical hardware. If it’s on a hypervisor, an attacker has another layer to go through before they can access the guests boot screen. If an attacker can access your VM through the hypervisors management app, you’re pretty well screwed anyway.

Disable Remote Root Logins

Review /etc/pam.d/login so we’re only permitting local root logins. By default this was setup that way.
Review /etc/security/access.conf . Make sure root logins are limited as much as possible. Un-comment rules that you want. I didn’t need to touch this.
Confirm which virtual consoles and text terminal devices you have by reviewing /etc/inittab then modify /etc/securetty by commenting out all the consoles you don’t need (all of them preferably). Or better just issue the following command to fill the file with nothing:

cat /dev/null > /etc/securetty

I back up this file before I do this.
Now test that you can’t log into any of the text terminals listed in /etc/inittab . Just try logging into the likes of your ESX/i vSphere guests console as root. You shouldn’t be able to now.

Make sure if your server is not physical hardware but a VM, then the hosts password is long and made up of a random mix of upper case, lower case, numbers and special characters.

Additional Resources

http://www.debian.org/doc/manuals/securing-debian-howto/ch4.en.html#s-restrict-console-login

SSH

My feeling after a lot of reading is that currently RSA with large keys (The default RSA size is 2048 bits) is a good option for key pair authentication. Personally I like to go for 4096, but with the current growth of processing power (following Moore’s law), 2048 should be good until about 2030. Update: I’m not so sure about the 2030 date for this now.

Create your key pair if you haven’t already and setup key pair authentication. Key-pair auth is more secure and allows you to log in without a password. Your pass-phrase should be stored in your keyring. You’ll just need to provide your local password once (each time you log into your local machine) when the keyring prompts for it. Of course your pass-phrase needs to be kept secret. If it’s compromised, it won’t matter how much you’ve invested into your hardening effort. To tighten security up considerably Make the necessary changes to your servers /etc/ssh/sshd_config file. Start with the changes I’ve listed here.
When you change things like setting up AllowUsers or any other potential changes that could lock you out of the server. It’s a good idea to be logged in via one shell when you exit another and test it. This way if you have locked yourself out, you’ll still be logged in on one shell to adjust the changes you’ve made. Unless you have a need for multiple users, lock it down to a single user. You can even lock it down to a single user from a specific host.
After a set of changes, issue the following restart command as root or sudo:

service ssh restart

You can check the status of the daemon with the following command:

service ssh status

Consider changing the port that SSH listens on. May slow down an attacker slightly. Consider whether it’s worth adding the extra characters to your SSH command. Consider keeping the port that sshd binds to below 1025 where only root can bind a process to.

We’ll need to tunnel SSH once the server is placed into the DMZ. I’ve discussed that in this post.

Additional Resources

Check SSH login attempts. As root or via sudo, type the following to see all failed login attempts:

cat /var/log/auth.log | grep 'sshd.*Invalid'

If you want to see successful logins, type the following:

cat /var/log/auth.log | grep 'sshd.*opened'

Consider installing and configuring denyhosts

Disable Boot Options

All the major hypervisors should provide a way to disable all boot options other than the device you will be booting from. VMware allows you to do this in vSphere Client.

Set BIOS passwords.

Lock Down the Mounting of Partitions

Getting started with your fstab.

Make a backup of your /etc/fstab before you make changes. I ended up needing this later. Read the man page for fstab and also the options section in the mount man page. The Linux File System Hierarchy (FSH) documentation is worth consulting also for directory usages.
Add the noexec mount option to /tmp but not /var because executable shell scripts such as pre, post and removal reside within /var/lib/dpkg/info .
You can also add the nodev nosuid options.
You can add the nodev option to /var, /usr, /opt, /home also.
You can also add the nosuid option to /home .
You can add ro to /usr

To add mount options nosuid,noexec to /var/tmp, /var/log, /var/account, we need to bind the target mount onto an existing directory. The following procedure details how to do this for /var/tmp. As usual, you can do all of this without a reboot. This way you can modify until your hearts content, then be confident that a reboot will not destroy anything or lock you out of your system.
Your /etc/fstab unmounted mounts can be tested like this

sudo mount -a

Then check the difference with

mount

mount options can be set up on a directory by directory basis for finer grained control. For example my /var mount in my /etc/fstab may look like this:

UUID=<block device ID goes here> /var ext4 defaults,nodev 0 2

Then add another line below that in your /etc/fstab that looks like this:

/var /var/tmp none nosuid,noexec,bind 0 2

The file system type above should be specified as none (as stated in the “The bind mounts” section of the mount man page http://man.he.net/man8/mount). The bind option binds the mount. There was a bug with the suidperl package in debian where setting nosuid created an insecurity. suidperl is no longer available in debian.

If you want this to take affect before a reboot, execute the following command:

sudo mount --bind /var/tmp /var/tmp

Then to pickup the new options from /etc/fstab:

sudo mount -o remount /var/tmp

For further details consult the remount option of the mount man page.

At any point you can check the options that you have your directories mounted as, by issuing the following command:

mount

You can test this by putting a script in /var and copying it to /var/tmp. Then try running each of them. Of course the executable bits should be on. You should only be able to run the one that is in the directory mounted without the noexec option. My file “kimsTest” looks like this:

#!/bin/sh
echo "Testing testing testing kim"

Then I…

myuser@myserver:/var$ ./kimsTest
Testing testing testing kim
myuser@myserver:/var$ ./tmp/kimsTest
-bash: ./tmp/kimsTest: Permission denied

You can set the same options on the other /var sub-directories (not /var/lib/dpkg/info).

Enable read-only / mount

There are some contradictions on /run/shm size allocation. Increase the size vs Don’t increase the size

Additional Resources

Work Around for Apt Executing Packages from /tmp

Disable Services we Don’t Need

RPC portmapper

dpkg-query -l '*portmap*'

portmap is not installed by default, so we don’t need to remove it.

Exim

dpkg-query -l '*exim*'

Exim4 is installed.
You can see from the netstat output below (in the “Remove Services” area) that exim4 is listening on localhost and it’s not publicly accessible. Nmap confirms this, but we don’t need it, so lets disable it. We should probably be using ss too.

When a run level is entered, init executes the target files that start with k with a single argument of stop, followed with the files that start with s with a single argument of start. So by renaming /etc/rc2.d/s15exim4 to /etc/rc2.d/k15exim4 you’re causing init to run the service with the stop argument when it moves to run level 2. Just out of interest sake, the scripts at the end of the links with the lower numbers are executed before scripts at the end of links with the higher two digit numbers. Now go ahead and check the directories for run levels 3-5 as well and do the same. You’ll notice that all the links in /etc/rc0.d (which are the links executed on system halt) start with ‘K’. Making sense?

Follow up with

sudo netstat -tlpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0: 0.0.0.0:* LISTEN 1910/sshd
tcp6 0 0 ::: :::* LISTEN 1910/sshd

And that’s all we should see.

Additional resources for the above

Disable Network Information Service (NIS). NIS lets several machines in a network share the same account information, such as the password file (Allows password sharing between machines). Originally known as Yellow Pages (YP). If you needed centralised authentication for multiple machines, you could set-up an LDAP server and configure PAM on your machines in order to contact the LDAP server for user authentication. We have no need for distributed authentication on our web server at this stage.

dpkg-query -l '*nis*'

Nis is not installed by default, so we don’t need to remove it.

Additional resources for the above

Remove Services

First thing I did here was run nmap from my laptop

nmap -p 0-65535 <serverImConfiguring>
PORT STATE SERVICE
23/tcp filtered telnet
111/tcp open rpcbind
/tcp open

Now because I’m using a non default port for SSH, nmap thinks some other service is listening. Although I’m sure if I was a bad guy and really wanted to find out what was listening on that port it’d be fairly straight forward.

To obtain a list of currently running servers (determined by LISTEN) on our web server. Not forgetting that man is your friend.

sudo netstat -tap | grep LISTEN

or

sudo netstat -tlp

I also like to add the ‘n’ option to see the ports. This output was created before I had disabled exim4 as detailed above.

tcp 0 0 *:sunrpc *:* LISTEN 1498/rpcbind
tcp 0 0 localhost:smtp *:* LISTEN 2311/exim4
tcp 0 0 *:57243 *.* LISTEN 1529/rpc.statd
tcp 0 0 *: *:* LISTEN 2247/sshd
tcp6 0 0 [::]:sunrpc [::]:* LISTEN 1498/rpcbind
tcp6 0 0 localhost:smtp [::]:* LISTEN 2311/exim4
tcp6 0 0 [::]:53309 [::]:* LISTEN 1529/rpc.statd
tcp6 0 0 [::]: [::]:* LISTEN 2247/sshd

Rpcbind

Here we see that sunrpc is listening on a port and was started by rpcbind with the PID of 1498.
Now Sun Remote Procedure Call is running on port 111 (also the portmapper port) netstat can tell you the port, confirmed with the nmap scan above. This is used by NFS and as we don’t need NFS as our server isn’t a file server, we can get rid of the rpcbind package.

dpkg-query -l '*rpc*'

Shows us that rpcbind is installed and gives us other details. Now if you’ve been following along with me and have made the /usr mount read only, some stuff will be left behind when we try to purge:

sudo apt-get purge rpcbind

Following are the outputs of interest:

The following packages will be REMOVED:
nfs-common* rpcbind*
0 upgraded, 0 newly installed, 2 to remove and 0 not upgraded.
Do you want to continue [Y/n]? y
Removing nfs-common ...
[ ok ] Stopping NFS common utilities: idmapd statd.
dpkg: error processing nfs-common (--purge):
cannot remove `/usr/share/man/man8/rpc.idmapd.8.gz': Read-only file system
Removing rpcbind ...
[ ok ] Stopping rpcbind daemon....
dpkg: error processing rpcbind (--purge):
cannot remove `/usr/share/doc/rpcbind/changelog.gz': Read-only file system
Errors were encountered while processing:
nfs-common
rpcbind
E: Sub-process /usr/bin/dpkg returned an error code (1)

Another

dpkg-query -l '*rpc*'

Will result in pH. That’s a desired action of (p)urge and a package status of (H)alf-installed.
Now the easiest thing to do here is rename your /etc/fstab to something else and rename the /etc/fstab you backed up before making changes to it back to /etc/fstab then because you know the fstab is good,

reboot

Then try the purge, dpkg-query and netstat commands again to make sure rpcbind is gone and of course no longer listening. I had to actually do the purge twice here as config files were left behind from the fist purge.

Also you can remove unused dependencies now after you get the following message:

The following packages were automatically installed and are no longer required:
libevent-2.0-5 libgssglue1 libnfsidmap2 libtirpc1
Use 'apt-get autoremove' to remove them.
The following packages will be REMOVED:
rpcbind*

sudo apt-get -s autoremove

Because I want to simulate what’s going to be removed because I”m paranoid and have made stupid mistakes with autoremove years ago and that pain has stuck with me. I autoremoved a meta-package which depended on many other packages. A subsequent autoremove for packages that had a sole dependency on the meta-package meant they would be removed. Yes it was a painful experience. /var/log/apt/history.log has your recent apt history. I used this to piece back together my system.

Then follow up with the real thing… Just remove the -s and run it again. Just remember, the less packages your system has the less code there is for an attacker to exploit.

Telnet

telnet installed:

dpkg-query -l '*telnet*'
sudo apt-get remove telnet

telnet gone:

dpkg-query -l '*telnet*'

Ftp

We’ve got scp, why would we want ftp?
ftp installed:

dpkg-query -l '*ftp*'
sudo apt-get remove ftp

ftp gone:

dpkg-query -l '*ftp*'

Don’t forget to swap your new fstab back and test that the mounts are mounted as you expect.

Secure Services

The following provide good guidance on securing what ever is left.

Scheduled Backups

Make sure all data and VM images are backed up routinely. Make sure you test that restoring your backups work. Backup system files and what ever else is important to you. There is a good selection of tools here to help. Also make sure you are backing up the entire VM if your machine is a virtual guest by export / import OVF files. I also like to backup all the VM files. Disk space is cheap. Is there such a thing as being too prepared for disaster? It’s just a matter of time before you’ll be calling on your backups.

Keep up to date

Consider whether it would make sense for you or your admin/s to set-up automatic updates and possibly upgrades. Start out the way you intend to go. Work out your strategy for keeping your system up to date and patched. There are many options here.

Logging, Alerting and Monitoring

From here on, I’ve made it less detailed and more about just getting you to think about things and ways in which you can improve your stance on security. Also if any of the offerings cost money to buy, I make note of it because this is the exception to my rule. Why? Because I prefer free software and especially when it’s Open Source FOSS.

Some of the following cross the “logging” boundaries, so in many cases it’s difficult to put them into categorical boxes.

Attackers like to try and cover their tracks by modifying information that’s distributed to the various log files. Make sure you know who has write access to these files and keep the list small. As a Sysadmin you need to read your log files often and familiarise yourself with them so you get used to what they should look like.

SWatch

Monitors “a” log file for each instance you run (or schedule), matches your defined patterns and acts. You can define different message types with different font styles. If you want to monitor a lot of log files, it’s going to be a bit messy.

Logcheck

Monitors system log files, emails anomalies to an administrator. Once installed it needs to be set-up to run periodically with cron. Not a bad we run down here. How to use and customise it. Man page and more docs here.

NewRelic

Is more of a performance monitoring tool than a security tool. It has free plans which are OK, It comes into it’s own in larger deployments. I’ve used this and it’s been useful for working out what was causing performance issues on the servers.

Advanced Web Statistics (AWStats)

Unlike NewRelic which is a Software as a Service (SaaS), AWStats is FOSS. It kind of fits a similar market space as NewRelic though, but also has Host Intrusion Prevention System (HIPS) features. Docs here.

Pingdom

Similar to NewRelic but not as feature rich. Update: Recently stumbled into Monit which is a better alternative. Free and open source. I’ve been writing about it here.

Multitail

Does what its name sounds like. Tails multiple log files at once. Provides realtime multi log file monitoring. Example here. Great for seeing strange happenings before an intruder has time to modify logs, if your watching them that is. Good for a single system if you’ve got a spare screen to throw on the wall.

PaperTrail

Targets a similar problem to MultiTail except that it collects logs from as many servers as you want and copies them off-site to PaperTrails service and aggregates them into a single easily searchable web interface. Allows you to set-up alerts on anything. Has a free plan, but you only get 100MB per month. The plans are reasonably cheap for the features it provides and can scale as you grow. I’ve used this and have found it to be excellent.

Logwatch

Monitors system logs. Not continuously, so they could be open to modification without you knowing, like SWatch and Logcheck from above. You can configure it to reduce the number of services that it analyses the logs of. It creates a report of what it finds based on your level of paranoia. It’s easy to set-up and get started though. Source and docs here.

Logrotate

Use logrotate to make sure your logs will be around long enough to examine them. Some usage examples here. Ships with Debian. It’s just a matter of applying any extra config.

Logstash

Targets a similar problem to logrotate, but goes a lot further in that it routes and has the ability to translate between protocols. Requires Java to be installed.

Fail2ban

Ban hosts that cause multiple authentication errors. or just email events. Of course you need to think about false positives here too. An attacker can spoof many IP addresses potentially causing them all to be banned, thus creating a DoS.

Rsyslog

Configure syslog to send copy of the most important data to a secure system. Mitigation for an attacker modifying the logs. See @ option in syslog.conf man page. Check the /etc/(r)syslog.conf file to determine where syslogd is logging various messages. Some important notes around syslog here, like locking down the users that can read and write to /var/log.

syslog-ng

Provides a lot more flexibility than just syslogd. Checkout the comprehensive feature-set.

Some Useful Commands

  • Checking who is currently logged in to your server and what they are doing with the who and w commands
  • Checking who has recently logged into your server with the last command
  • Checking which user has failed login attempts with the faillog command
  • Checking the most recent login of all users, or of a given user with the lastlog command. lastlog comes from the binary file /var/log/lastlog.

This, is a list of log files and their names/locations and purpose in life.

Host-based Intrusion Detection System (HIDS)

Tripwire

Is a HIDS that stores a good know state of vital system files of your choosing and can be set-up to notify an administrator upon change in the files. Tripwire stores cryptographic hashes (delta’s) in a database and compares them with the files it’s been configured to monitor changes on. Not a bad tutorial here. Most of what you’ll find with tripwire now are the commercial offerings.

RkHunter

A similar offering to Tripwire. It scans for rootkits, backdoors, checks on the network interfaces and local exploits by running tests such as:

  • MD5 hash changes
  • Files commonly created by root-kits
  • Wrong file permissions for binaries
  • Suspicious strings in kernel modules
  • Hidden files in system directories
  • Optionally scan within plain-text and binary files

Version 1.4.2 (24/02/2014) now checks ssh, sshd and telent (although you shouldn’t have telnet installed). This could be useful for mitigating non-root users running a modified sshd on a 1025-65535 port. You can run ad-hoc scans, then set them up to be run with cron. Debian Jessie has this release in it’s repository. Any Debian distro before Jessie is on 1.4.0-1 or earlier.

The latest version you can install for Linux Mint Qiana (17) and Rebecca (17.1) within the repositories is 1.4.0-3 (01/05/2012)

Change-log here.

Chkrootkit

It’s a good idea to run a couple of these types of scanners. Hopefully what one misses the other will not. Chkrootkit scans for many system programs, some of which are cron, crontab, date, echo, find, grep, su, ifconfig, init, login, ls, netstat, sshd, top and many more. All the usual targets for attackers to modify. You can specify if you don’t want them all scanned. Runs tests such as:

  • System binaries for rootkit modification
  • If the network interface is in promiscuous mode
  • lastlog deletions
  • wtmp and utmp deletions (logins, logouts)
  • Signs of LKM trojans
  • Quick and dirty strings replacement

Stealth

The idea of Stealth is to do a similar job as the above file integrity scanners, but to leave almost no sediments on the tested computer (called the client). A potential attacker therefore has no clue that Stealth is in fact scanning the integrity of its client files. Stealth is installed on a different machine (called the controller) and scans over SSH.

Ossec

Is a HIDS that also has some preventative features. This is a pretty comprehensive offering with a lot of great features.

Unhide

While not strictly a HIDS, this is quite a useful forensics tool for working with your system if you suspect it may have been compromised.

Unhide is a forensic tool to find hidden processes and TCP/UDP ports by rootkits / LKMs or by another hidden technique. Unhide runs in Unix/Linux and Windows Systems. It implements six main techniques.

  1. Compare /proc vs /bin/ps output
  2. Compare info gathered from /bin/ps with info gathered by walking thru the procfs. ONLY for unhide-linux version
  3. Compare info gathered from /bin/ps with info gathered from syscalls (syscall scanning)
  4. Full PIDs space ocupation (PIDs bruteforcing). ONLY for unhide-linux version
  5. Compare /bin/ps output vs /proc, procfs walking and syscall. ONLY for unhide-linux version. Reverse search, verify that all thread seen by ps are also seen in the kernel.
  6. Quick compare /proc, procfs walking and syscall vs /bin/ps output. ONLY for unhide-linux version. It’s about 20 times faster than tests 1+2+3 but maybe give more false positives.

It includes two utilities: unhide and unhide-tcp.

unhide-tcp identifies TCP/UDP ports that are listening but are not listed in /bin/netstat through brute forcing of all TCP/UDP ports available.

Can also be used by rkhunter in it’s daily scans. Unhide was number one in the top 10 toolswatch.org security tools pole

Web Application Firewalls (WAF’s)

which are just another part in the defense in depth model for web applications, get more specific in what they are trying to protect. They operate at the application layer, so they don’t have to deal with all the network traffic. They apply a set of rules to HTTP conversations. They can also be either Network or Host based and able to block attacks such as Cross Site Scripting (XSS), SQL injection.

ModSecurity

Is a mature and feature full WAF that is designed to work with such web servers as IIS, Apache2 and NGINX. Loads of documentation. They also look to be open to committers and challengers a-like. You can find the OWASP Core Rule Set (CRS) here to get you started which has the following:

  • HTTP Protocol Protection
  • Real-time Blacklist Lookups
  • HTTP Denial of Service Protections
  • Generic Web Attack Protection
  • Error Detection and Hiding

Or for about $500US a year you get the following rules:

  • Virtual Patching
  • IP Reputation
  • Web-based Malware Detection
  • Webshell/Backdoor Detection
  • Botnet Attack Detection
  • HTTP Denial of Service (DoS) Attack Detection
  • Anti-Virus Scanning of File Attachments

Fusker

for Node.js. Although doesn’t look like a lot is happening with this project currently. You could always fork it if you wanted to extend.

The state of the Node.js echosystem in terms of security is pretty poor, which is something I’d like to invest time into.

Fire-walling

This is one of the last things you should look at when hardening an internet facing or parameterless system. Why? Because each machine should be hard enough that it doesn’t need a firewall to cover it like a blanket with services underneath being soft and vulnerable. Rather all the services should be either un-exposed or patched and securely configured.

Most of the servers and workstations I’ve been responsible for over the last few years I’ve administered as though there was no firewall and they were open to the internet. Most networks are reasonably easy to penetrate, so we really need to think of the machines behind them as being open to the internet. This is what De-perimeterisation (the concept initialised by the Jericho Forum) is all about.

Some thoughts on firewall logging.

Keep your eye on nftables too, it’s looking good!

Additional Resources

Just keep in mind the above links are quite old. A lot of it’s still relevant though.

Machine Now Ready for DMZ

Confirm DMZ has

  • Network Intrusion Detection System (NIDS), Network Intrusion Prevention System (NIPS) installed and configured. Snort is a pretty good option for the IDS part, although with some work Snort can help with the Prevention also.
  • incoming access from your LAN or where ever you plan on administering it from
  • rules for outgoing and incoming access to/from LAN, WAN tightly filtered.

Additional Web Server Preparation

  • setup and configure soft web server
  • setup and configure caching proxy. Ex:
    • node-http-proxy
    • TinyProxy
    • Varnish
    • nginx
  • deploy application files
  • Hopefully you’ve been baking security into your web app right from the start. This is an essential part of defense in depth. Rather than having your application completely rely on other entities to protect it, it should also be standing up for itself and understanding when it’s under attack and actually fighting back.
  • set static IP address
  • double check that the only open ports on the web server are 80 and what ever you’ve chosen for SSH.
  • setup SSH tunnel
  • decide on and document VM backup strategy and set it up.

Machine Now In DMZ

Setup your CNAME or what ever type of DNS record you’re using.

Now remember, keeping any machine on (not just the internet, but any) a network requires constant consideration and effort in keeping the system as secure as possible.

Work through using the likes of harden and Lynis for your server and harden-surveillance for monitoring your network.

Consider combining “Port Scan Attack Detector” (psad) with fwsnort and Snort.

Hack your own server and find the holes before someone else does. If you’re not already familiar with the tricks of how systems on the internet get attacked read up on the “Attacks and Threats” Run OpenVAS, Run Web Vulnerability Scanners

From here on is in scope for other blog posts.

Procurement & Config of Sun Fire V240 & ALOM

October 25, 2014

This is the sequence of events I took to prepare a Sun Fire V240 for hosting pfSense which is a free and open source FreeBSD based enterprise grade routing solution for a client of mine.

Recently I was tasked with setting up a network with what I considered to be enterprise grade hardware and software as cheaply as possible. When I take on these sorts of tasks, security is forefront in my mind, so I often look toward components that are as open as possible and that don’t sport any known (to me at least) back-doors and are able to be easily upgraded and patched at little to no cost.

A requirement was clean shut-downs on power failure events at least for the critical servers.

Procured Kit

  1. APC Smart-UPS 5000 with batteries in good condition. Worth a little under $6k if you’re buying new. I wouldn’t buy new. If you shop around, these can be picked up at a fraction of that cost. From my experience the APC kit is some of the best UPS gear available.
    APC Smart-UPS 5000
  2. AP9630 UPS network management card $92 new. Most of the details around setting these UPS’s up I’ve already posted on. If you search my blog for “APC UPS” you’ll find it.
    APC AP9630
  3. Enterprise grade router/firewall:
    Sun Fire V240 (RISC architecture). 2 x UltraSparc-IIIi 1.5Ghz CPU. 4Gbit on-board Ethernet ports. Lights-out management port. 4GB RAM. 2U. Dual redundant PSU’s. 2 x 72GB Hot Swap 10k SCSI HDD’s. With rack mount rails. Currently going for around $1.5k on Ebay. Price paid: $160 incl shipping. I doubt you’d find anything of these specifications off the shelf for under a $1000. This is a lot of server for a very small amount of money.
    Sun Fire V240
  4. Firmware: pfSense. Free and open source.

Planning

As part of my planning I evaluated (again) whether or not free software routing solutions are actually up to the task of the enterprise. My research led me to believe some were… based on others that had already been down this route ( PTP 😉 ). Openness is a biggie for me. I like to know that eyes are on the software rather than it being closed up in a proprietary package.

I evaluated m0n0Wall, ipCop (Linux based), smoothwall and pfSense. pfSense had been used in quite a few large environments successfully. When I had made my decision on the firmware to use, I went through the hardware requirements and of course started looking for high quality second-hand gear.

For the router hardware I was going to need at lease 1GHz CPU as I wanted to run Snort as my IDS/IPS. PCI-X or PCI-e network adapters (which of course I didn’t need to worry about with the Sun Fire server). Snort needs 512MB RAM minimum. Preferably at least 1GB.

Gaining Access to the Sun Fire V240

Now I had no idea of how the previous owner had setup the configuration of the ALOM (Advanced Lights Out Management). In fact I hadn’t administered a Sun Fire server before at all. On page 11 of the Sun FireTM V210 and V240 Servers Getting Started Guide it states the following:

The system console is directed to ALOM by default and is configured to show server console information on startup.
ALOM enables you to monitor and control your server over either a serial
connection (using the SERIAL MGT port), or Ethernet connection (using the NET MGT port).
For information about configuring an Ethernet connection, refer to the Sun Advanced Lights Out Manager Software User’s Guide.” The NET MGT port can also be disabled and in my case it turned out it was, but I’ll get to that later. I didn’t have a spare DB-9 to RJ-45 adapter lying around to wire it up and connect to the SERIAL MGT port.

Sun Fire V240 rear

Telnet?

(but didn’t get that far)

Since I was going to go down the path of trying to connect to the ALOM console via the NET MGT Ethernet port, I thought telnet would probably be the path of least resistance.

Page 10 of the “Sun Advanced Lights Outs Manager Software User’s Guide” stated the following:

The 10-Mbyte Ethernet port enables you to access ALOM from within your company
network. You can connect to ALOM remotely using any standard Telnet client”. On the V240, the
ALOM Ethernet port is referred to as the NET MGT port.

Using a laptop with Kali Linux installed (because it has lots of great tools for network reconnaissance), Running

ethtool eth0

told me that my NIC supported:
10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full

Wireshark?

Tried connecting directly to the NET MGT port with wireshark running on my laptop. Didn’t get any packets from the device. At the time I thought it may have been because my laptop’s NIC was using 100baseT, but later on I found out that the NET MGT port was disabled.

Tried pinging my broadcast address ping -b 255.255.255.255 then checked my arp table arp -a. No results that looked like what I was looking for. Of course this strategy would have taken quite some time to complete… and in my case it would have yielded no results anyway.

NMap?

I started with the private IPv4 address spaces. Using Wi-fi on my Kali box, tried the 16 bit block:

nmap -sn 192.168.*.*

Got a false positive of a cable modem. How did I work out that it was a false positive?

nmap -A <falsePositiveIPOfCableModem> # Gave me the model and everything I needed to know about the device to rule it out.

Next up the 20 bit block

nmap -sn 172.16.0.0/12
Nmap done: 1048576 IP addresses (0 hosts up) scanned in 108670.97 seconds

In earlier releases of nmap the -sn switch was known as -sP

I decided I needed to try and speed up the scan, so I connected directly to the V240 NET MGT port with a Cat5 patch cable (ethtool told me my laptop’s NIC had MDI-X on (force crossover mode)) and made sure my network card supported 10baseT which the “Sun Advanced Lights Outs Manager Software User’s Guide” told me it needed for the NET MGT port. Turns out the NET MGT port didn’t support 10baseT. Details a bit further down.

Added a static IP address to the /etc/network/interfaces. Currently it looked like:

auto lo
iface lo inet loopback

auto wlan0
iface wlan0 inet dhcp

So I commented out the auto wlan0 and iface wlan0 inet dhcp and added the following:

auto eth0
iface eth0 inet static
address 10.1.1.6
netmask 255.255.255.0
broadcast 10.1.1.255
#gateway 10.1.1.1 # Make sure you don't add a gateway, as we're connecting directly to the V240

followed by:

service networking restart

then changed my /etc/NetworkManager/NetworkManager.conf
managed=true to be managed=false
So Network manager didn’t keep interfering with my interfaces.

I followed this with a

service network-manager restart

followed with ifconfig to make sure my network interface was using the correct IP address, netmask and broadcast. It wasn’t, so…

ifdown eth0
ifup eth0
ifconfig

Success, it now was.

Now to make sure my network card was communicating in a manner that the V240’s NET MGT port would understand.

Using ethtool

ethtool eth0

told me 10baseT was supported, but it also told me my current speed was 100Bb/s. So I tried changing the speed with

ethtool -s eth0 speed 10

and received Cannot advertise speed 10. So made the following temporary changes as they’ll be lost on reboot. Changed the duplex… Ran the following:

ethtool -s eth0 speed 10 duplex half

Now with a:

ethtool eth0

I got:

Speed: unknown!
Duplex: Unknown! (255)

So turned the auto negotiation off:

ethtool -s eth0 speed 10 duplex half autoneg off

Now with a:

ethtool eth0

I got:

Speed: 10Mb/s
Duplex: Half
Auto-negotiation: off
#and some other settings.

Some useful ethtool resources:

With these settings the NET MGT port didn’t have it’s green link led on. So I kept playing with the settings. Turns out it would only work with speed 100 duplex full contrary to page 10 of the “Sun Advanced Lights Out Manager Software User’s Guide”
These were the settings that gave me link:

Supported pause frame use: No #Don't think I fiddled with this.
Supports auto-negotiation: Yes
Advertised link modes: Not reported #Don't think I fiddled with this.
Advertised pause frame use: Symmetric #Don't think I fiddled with this.
Advertised auto-negotiation: No
Speed: 100Mb/s
Duplex: full
Port: Twisted Pair #Don't think I fiddled with this.
PHYAD: 1 #Don't think I fiddled with this.
Transceiver: internal #Don't think I fiddled with this.
Auto-negotiation: off
MDI-X: on
Supports Wake-on: g #Don't think I fiddled with this.
Wake-on: d #Don't think I fiddled with this.
Current message level: 0x000000ff (255)
drv prove link timer ifdown ifup rx_err tx_err
Link detected: yes

I was now confident that if the Sun Fire V240 NET MGT port was enabled, we’d find it’s IP address if it was using one from the private space. It was time to try the last and largest private address space. Oh, I also used wireshark to make sure nmap was doing what I expected on my laptop when I ran:

nmap -v -sn 10.0.0.0/8

I was a little confused to start with as nmap told me Scanning 4096 hosts I soon realised after checking the CIDR (Classless InterDomain Routing) and by the output nmap produced, that nmap was doing the scanning in chunks. As there was going to be a lot of results, I setup the output to files:

nmap -v -sn -oA 'scan-%Y-%m-%d_%H-%M 10.0.0.0/8

This produces the output in all three formats as discussed here.

SERIAL MGT Port?

This private address range was going to take a few days to scan, so I decided to have a poke at the SERIAL MGT port on the Sun Fire V240.

To use the SERIAL MGT port, a RJ-45 patch cable connected to a DB-9 adapter ($4.50 from globalpc) is required Unless you get the official Sun adaptor “530-3100-01”, or still have the one that came in the new box. So I splashed out and went with the $4.50 option. It cost me more in gas to get to the shop than buy the part. I Wired it up according to page 25 of the “Sun Fire V210 and V240 Servers Installation Guide“.

RJ-45 to DB-9 Adapter Crossovers
SERIAL MGT Port Adapter (DB-9) Pin
1 (RTS) 8 (CTS)
2 (DTR) 6 (DSR)
3 (TXD) 2 (RXD)
4 (Signal Ground) 5 (Signal Ground)
5 (Signal Ground) 5 (Signal Ground)
6 (RXD) 3 (TXD)
7 (DSR) 4 (DTR)
8 (CTS) 7 (RTS)

Red wire in with green.

RJ45-DB9 RJ45

Installed minicom and setserial and did pretty much the same as I did here. Plugged the console cable in and tried to establish a connection.

Then found that by default ALOM only communicates through the SERIAL MGT port at startup (of ALOM I thought), but it seems that at power on of the server also.

At the {1} ok prompt, I typed #. (that’s hash followed with dot) to escape from the system console sc>

I then entered the showsc command and found that the MGT NET port was disabled.
I then ran a

usershow

to see which user accounts existed and was prompted to set a password for the admin user.
When you connect to ALOM for the first time, you are automatically connected as the admin account.“.
So obviously the seller of the system reset ALOM.

SettingAdminPassword

Also audited the user accounts, and the details on the permission levels are here.

Ran the following script. A nice little dialog from Ramesh here (see step 4) too.

setupsc
  • Turned NET MGT port on
  • Changed the default if_connection from none to ssh
  • Answered no to email alerts (only for logged in users)
  • Yes to configure the network interfaces
  • No to DHCP
  • Entered the IP address for the NET MGT port
  • Entered the netmask for the NET MGT port
  • Entered the gateway for the Net Mgt port
  • Should powerstate memory be enabled [y]? y
  • Enabled power on sequencing

Then we need to restart the ALOM to apply the new settings.

resetsc -y

If you still have minicom running, it’ll show you what happens during the boot sequence and then present you with a login prompt.

Extra Resources

SSH

At this point I plugged the Ethernet cable from my test switch (10 Mbit/s capable) back into the NET MGT port of the Sun Fire V240 and tested that ALOM was responding on the IP address that I set the NET MGT port to.

ping <myNetMgtIP>

It was answering. So I attempted to SSH in on a different machine.

ssh admin@<myNetMgtIP>

I was presented with the hosts key fingerprint

The authenticity of host <myNetMgtIP> (<myNetMgtIP>)' can't be established.
RSA key fingerprint is <myExistingHostKeyInHex>.
Are you sure you want to continue connecting (yes/no)?

I wanted to know I was connecting to what I thought I was connecting to, so answered no.
Then in minicom I queried the hosts key fingerprint

ssh-keygen -l -t rsa

I was provided with the key fingerprint that matched what I was presented with when I attempted to SSH, so I new I was actually communicating with the server I thought I was.

I then regenerated the hosts key fingerprint

ssh-keygen -r -t rsa

and was provided with the new key. A restart of the SSH daemon is required to load the new host key.

sc> restartssh

Then SSH in. Confirm when prompted that the host key matches the newly provided key.

ssh admin@<myNetMgtIP>
The authenticity of host <myNetMgtIP> (<myNetMgtIP>)' can't be established.
RSA key fingerprint is <myNewHostKeyInHex>.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '<myNetMgtIP>' (RSA) to the list of known hosts.

Copyright 2009 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.

Sun(tm) Advanced Lights Out Manager <versionHere> ()

Please login: admin
Please Enter password: *********

sc>

We’re in!

At any time for a list of commands, you can type help.

logout
Connection to <myNetMgtIP> closed.

We’re out!

Automating Specification by Example for .NET Web Applications

February 22, 2014

If you or your organisation:

  1. are/is constrained to running your .NET tests (unit, acceptance) on-site rather than in the cloud
  2. would like some guidance on how to set-up Continuous Integration

read on.

Introduction

Purpose

Remember, an acceptance test system as a tool is only as good as the specification provided by it’s humans. The most important ingredients there-for is the relationships between the people creating the tests and the interactions performed by those people. Or as the Agile Manifesto states: Value “Individuals and interactions over processes and tools”. In order for an acceptance test system to be successful, the relationships of the Developers creating the increment and the interactions between them and the stake holders must be in good shape first. Once this is in order, you can take the next step and find some tools that will assist in creating working software that does what the stake holders want it to do.

It’s my intention that the following details will help you to create a system that automates “Specification by Example”.

The purpose of providing an automated Specification by Example Implementation, A.K.A Automated Acceptance Test System, is clearly explained here.

Do not fall into the trap of inverting the test triangle. Instead invest where it matters.

Scope

Create a system that can be triggered from

  1. Every developers workstation
  2. A build on the build machine, preferably from a best of bread build tool. TFS is not a best of bread build tool and if you want to get serious about Continuous Integration (CI), nightly builds, continuous deployment, I’d recommend not going down the path of TFS. Even Microsoft uses Git. Doesn’t that tell you something? Do you see TFS here? Last time I evaluated build tools, Jenkins previously named Hudson came out on top.

jenkins

The system will include

  1. An acceptance test framework that will run all the acceptance tests
  2. A Unit test framework. UI tests need to be run in parallel on a collection of VM’s (See the section on supported browsers for why). There are three immediately obvious approaches we could take here.
    1. We could try and rely on a unit test framework to distribute the tests. MSTest 2012 doesn’t provide the ability to run tests in parallel, but 2010 does. In order to have 2012 run tests in parallel, you can force it to use the 2012 test settings file. Only a maximum of 5 tests can be run concurrently though. Not a great option, considering it’s not going to be supported going forward.
    2.  My ParallelBrowser. If this link is not active and you’re interested in this, contact me.
    3. PNUnit. An example of how this works is here under the “PNunit Framework for writing selenium test cases” heading. I wrote the ParallelBrowser before Selenium had good support for running the same tests on multiple supported browsers. Both my ParallelBrowser and this option are reasonable options, but I’d go for the latter now. This way someone else can maintain the parallel aspect. As unless people are interested in ParallelBrowser I won’t be doing any further work on it.
  3. A Web User Interface Test Framework that will be driven by the acceptance test framework. Selenium in this case.
  4. A set of tests that run Selenium tests. These will of course need to be thread-safe.
  5. As per the Supported Browsers section, a collection of VM’s with our supported browsers installed.
    1. Each with a standalone selenium server setup with a role of webdriver. Details further on.
  6. A stand-alone selenium server setup with a role of hub

High Level Flow

Many organisations bound to .NET seem to be locked into using sub-standard tooling like TFS for their build. If you are in this predicament and can not break free, I’d suggest once all the unit tests, integration tests have run, then have the build kick off a psake script to:

  1. Clean out the existing target web app
  2. Deploy the newly built and tested web app
  3. Drop the database
  4. Create database by using latest DDL and DML scripts pulled from source control
  5. Apply any specific configurations
  6. Stop and start the target web server
  7. Run the acceptance tests which will include any Web UI tests.

If it’s within your power to choose a real CI Tool to run in-house, there are a handful of very solid contenders. A good proportion of which are free and open source.

Audience

Who ever is setting up the system. Often a developer or two. It’s important to make sure more than one person knows how it all hangs together, otherwise you have a single point of failure.

Chosen Tools

Evaluation Criterion I used

  • Who is the creator? I favour teams rather than individuals, as individuals move on often leaving projects stranded?
  • Does it do what you need it to do?
  • Does it suite the way you and your team want to work?
  • Does it integrate well with all of your other chosen components? This is based on communicating with those that have used the offerings more so than using Proof Of Concepts (POC).
  • Works with the versions of dependencies you currently use.
  • Cost in money. Is it free? Are there catches once you get further down the road? Usually open source projects are marketed as is. No catches
  • Cost in time. Is the set-up painful? Customisation feedback? Upgrade feedback?
  • How well does it appear to be supported? What do the users say?
  • Documentation. Is there any / much? What is its quality?
  • Community. Does it have an active one? Are the users getting their questions answered satisfactorily? Why are the unhappy users unhappy (do they have valid reasons).
  • Release schedule. How often are releases being made? When was the last release?
  • Intuition. How does it feel. If you have experience in making these sorts of choices, lean on it. Believe it or not, this should probably be No. 1

The following tools have been my choice based on the above criterion.

Acceptance Test Framework

The following offerings are all free and open source.

If you’re not using User Stories and/or Test Conditions, the context/specification offerings provide greater flexibility than the xBehave style frameworks. As most Scrum teams use User Stories for their Product Backlog items and drive their acceptance tests with test conditions, xBehave offerings are a great choice. In saying that, there is probably no reason why both couldn’t be used where it makes sense to do so. In this section I’ve provided the results of evaluating the current xSpec and xBehave offerings for .NET ordered by best first for the categories.

xBehave (test conditions)

SpecFlow

specflow

  • Sourcecode: https://github.com/techtalk/SpecFlow/
  • Age: Over 4 years
  • Actively maintained: Yes
  • Large number of active committers
  • Community: Lively
  • Visual Studio Plug-in has been downloaded 70 times as many times as NBehave
  • Documentation: Excellent
  • Integrates well with Selenium (I’ve setup a couple of systems using SpecFlow and it’s been a joy to work with). The stake holders loved the visibility it provided too. I discussed it here in a recent presentation.
NBehave
  • Not a lot of activity
  • Only two committers
StoryQ
  • Only two coordinators
  • Well established framework

xSpec (context/specification)

Machine.Specification (MSpec)
NSpec

Web User Interface Test Framework

selenium

For me when I look at this category of tools for .NET, Selenium is always at the top and it just keeps getting better. If anyone has any questions around Selenium, feel free to contact me or leave a comment on this post. I can’t guarantee I’ll have the answer, but I’ll try. All the documentation can be found here. I would recommend installing the Selenium IDE for initially recording tests and be sure to check-out the IDE plug-ins. All the documentation you’ll need for the IDE is here. Once you get familiar with the code it generates, you will not use it much. I would recommend using the newer Web drivers rather than the selenium server by itself. The user group is very active and looks like a good place to ask questions also. Although I haven’t needed to as there is a huge amount of documentation that’s great.

The tools I would use are detailed here. Specifically we would be using

  1. Selenium 2 (aka WebDriver)
  2. The IDE for recording tests initially
  3. Selenium Server which is used by WebDriver and RC (now considered legacy) now includes built-in grid capabilities.

Supported Browsers

What I’ve done in the past is have each of our supported versions from each supported browser vendor installed on a single VM. So each VM has all the vendors browsers installed, but just a single version obviously.

Mid Level Flow

These are the same points listed above under “High Level Flow

1. Build Kicks off PSake Script

psake

The choice to use PSake over the likes of NAant, Rake and the other build scripting languages is reasonably straight forward for me. PSake (PowerShell build scripting language) gives us access to the full .NET environment. NAnt with all it’s angle brackets, was never a very nice scripting language to use. Rake is excellent and a possible option if you have ruby installed. If you don’t, why install it if you have .NET? There are many resources for PowerShell on the inter-webs. The wiki for PSake is good.

In the case where you may have a TFS Build run, I would suggest once all the unit tests and integration tests have run, then the build kicks off a possibly pre-build and post-build psake script to perform the following operations. This is how you do this. Oh, before you try to actually run a PSake script, download and import the module, or install the NuGet package. So once you have your PSake scripts running, just start adding PowerShell scripts to do the following work. PSake is just syntactic sugar around PowerShell, so anything you can do with PS, you can do with PSake.

2. Clean out the existing target web application

Using your PSaki script, use the Web Deploy cmdlets. You will find everything you need here for it. You can also install the NuGet package.

3. Deploy the newly built and unit tested web application

As above, just use the Web Deploy cmdlets.

4. Drop the database

As above, just use the Web Deploy cmdlets.

5. Create database by using latest DDL and DML scripts pulled from source control

Database update via Application

Kind of related, but not specific to CI.

Depending on your needs, there are quite a few ways you could do this.

One way of doing this is to have your application utilise a library that determines which version of the database the application needs and be able to update the database accordingly. This library would use similar or the same upgrade scripts that we would use in this test process.

Your applications should create (if non existent) and update database on run. So all the DDL, DML code per database lives in a library. Each application that uses a specific database, references the databases DDL code library. Script all stored procedures, views, functions, triggers they’re recreated as part of a deployment scrip.

When the application is deployed, and the database created or updated, anything that must be there for the application to run out of the box should be part of the scripts, and of course versioned. This includes the part of our data that is constant or configuration data. Tables, stored procedures, views, functions and triggers. For the variable part of your data, you will need a synthetic data generation plan for testing.

Database Process for Versioning

Also related, but not specific to CI.

DBA, Devs, Product Owner and consultants must be aware of the process.

When any schema, constant data, configuration data, test data is updated… the (version controlled) scripts must also be updated, else the updates will get overwritten.

As part of the nightly build, if your supporting multiple versions of your application, you could also hydrate the collection of database versions, then run the appropriate upgrade scripts against each one, to verify the upgrades work. If any don’t, the build fails.

Create set of well defined processes that:

  1. In most cases, looks after itself
  2. Upgrades existing databases if they are not on the latest version, to the latest version
  3. Creates databases for those applications that don’t have a database
  4. Informs the user on deployment if the database is corrupt, or can not be upgraded
  5. Outlines who is responsible for, and who may update the DDL and DML scripts for your projects
  6. Clearly documents that any changes made to any databases by un-authorised personal will more than likely be overwritten.

A User Story for this might look something like the following:

As the team, we need to create a set of well defined processes that clearly outline what is required in regards to setting up the development teams database versioning, creation, upgrade systems and processes strategy for our organisations databases. So that all team personal are aware of the benefits and dangers of making changes to the databases, and understand the change process.

Possibly useful tools

1. DB Ghost
2. http://www.red-gate.com/products/sql-development/sql-source-control/index-2
3. http://www.sqlaccessories.com/SQL_Data_Examiner/

6. Apply any specific configurations

As above, just use the Web Deploy cmdlets.

7. Stop and start the target web server

As above, just use the Web Deploy cmdlets.

8. Run the acceptance tests which will include any Web UI tests

As above, just use the Web Deploy cmdlets.

  1. Start each VM that hosts a set of browsers you want to use to farm your tests out to. From memory, you do not need to start each browser. There are of course many ways to do this. PS provides the following cmdlets Start-VM and Stop-VM. These would be my first options.
  2. Start the selenium standalone server. All details found here. Or just work through the “Distributed Testing with Selenium Grid” chapter until you get to the “Creating and executing Selenium script in parallel with TestNG” heading, at which point switch to this documentation to replace TestNG with PNUnit.

If I’ve failed to explain anything in enough detail for you, drop me a message below and I’ll do my best to help 🙂

Reassembly of the Eee PC 901

June 8, 2013

This is a follow on from “Upgrade Linux Eee PC 901 4GB SSD

As usual, this is just the previous section but in reverse. Hopefully you won’t have any screws left when it’s back together.

Now the reassembly is just as hard if not harder. You have to be as careful as possible not to put your statically charged body parts all over the circuitry of the Eee PC.
I found that the screw holes on the new SSD didn’t line up with the holes on the motherboard. So if you have a very small round file you can file them bigger, or some very small side cutters, clip them bigger. That’s what I did as I didn’t have a file small enough to get through the holes. Now the screw heads wouldn’t hold the SSD down any more. Zip ties to the rescue.

Now when you put the motherboard on the plastic base, you need to plug the cable for the bottom row of LED’s back in and the CPU fan (you can see these in my previous post). Then put the top two screws back in.

Next step is to plug the mouse cable into its socket (highlighted by the red box below). this is really fiddly.

Plugging the touch pad in

Then plug the cable that takes the signal from the 4 top buttons above the keyboard into its socket.

Next is to plug the keyboard ribbon back in to its socket. I found the method of least friction to be through the top piece of sheet metal. What I did here was to hold the ribbon down flat and keep a little pressure on top of the ribbon and directed into the socket while with long nose pliers a push on each side in sequence until the ribbon was all the way in. Again this is quite fiddly.

PlugInKeyboardRibbon

Put the top half of the plastic chassis back on in reverse to how it was removed… top first, then work your way down the sides. Put the six screws back into the panel under the keyboard. Flip the device, install the 16GB SSD and the RAM module. Put all the screws back in and that should be it.

Upgrade Linux Eee PC 901 4GB SSD

May 25, 2013

Update 01/09/2013 Looks like the SX300 mSATA SSD is incompatible with the Eee PC 901.

That’s the device with the 4GB primary SSD and the 16GB secondary SSD.

In looking at Offensive Securities new improved OS Kali, I was keen to take it for a spin. I’ll go over that in a subsequent post. This meant I’d have to find a suitable host for Kali. Kali can be installed on a myriad of platforms. It just so happens that the Eee PC 901 is an excellent device for the purpose, providing it has enough space (8GB incl the 300 security tools it ships with) which it currently didn’t.

For a while now I’ve had a couple of old Eee Pc 901’s I purchased a few years ago… with the 4GB Phison Primary mSATA SSD 50mm x 32mm and the 16GB Phison Secondary SATA SSD (2.5″ (70mm) long). I had read and assumed it to be correct that the mini card, that’s the smaller one in form factor and storage capacity was soldered onto the motherboard and that swapping it out was clouded with mystery and danger.

Top of the 50mm and 70mm SSD’s

PCIe SATA SSD

Bottom of the 50mm and 70mm SSD’s

PCIe SATA SSD

First thing I did was to upgrade the BIOS.
ROM: here.
HowTo: here. I used the Easy Flash.

Now the 16GB is easily replaceable as it’s on the bottom of the notebook by the RAM, but I don’t care about that drive. The boot order can’t be changed from the 4GB to the 16GB which is disappointing. So I had to replace the 4GB with a larger drive. As SSD’s are getting faster now also, we notice a significant speed increase. The Atom CPU’s do really well for their size and age, and with the 901’s even though they came out with solid state drives, the newer ones are considerably faster.
For a lot of different applications, the 4GB is just too small. I’ve been waiting for a purpose to use these devices. Now after a bit of research, I found out that the 4GB can actually be replaced without any soldering as I had initially thought necessary.
So that’s what I did. The Epc 901’s are great hacking devices. Light, still reasonably fast CPU’s for what they are and the devices can take a bit of a beating and with the SSD’s you don’t have to worry about HDD’s crashing.

Dismantle the 901

As always, remove all power sources first. That means remove the power cord and the battery. Remove all 13 screws from the underside of the chassis. This includes removing the RAM and 16GB plastic cover and removing the two screws holding in the 16GB SSD. Remove the 16GB SSD and the 1GB (unless you’ve upgraded to 2GB which I had) RAM module. Now of course I didn’t care about voiding the warranty as that was voided years ago.
Now turn the device over and you need to prise the keyboard off.

Removing Eee PC 901 Keyboard

The (3) plastic clips holding the keyboard on are along the top of it. I used a steak knife on each side, but you can use a jewellers screwdriver if you have one (maybe a better option, but the knives worked for me). When you’re pulling the keyboard up at the top, try and keep the pressure on the top edge of the keyboard pushing toward the bottom edge of the keyboard, this helps the clips release. Once these are released, the keyboard can slide up. You can see the 3 retainers on the bottom edge of the keyboard in the below image.

Eee PC 901 keyboard chassis

As you can see in the  above image, there are another six screws holding the top metal piece down through the motherboard I think and onto the bottom half of the plastic chassis. Remove these. Including the one that has the sticker that voids your warranty covering it… providing you don’t care about that?
Now remove the top half of the plastic chassis from the plastic base. There are some clips along each side, which release reasonably easy with a bit of jiggling. Pull the bottom (nearest to yourself) of the top half of the plastic chassis up. When you have enough room to get in between the top and the base with a small set of long nose pliers, pull the keyboard ribbon out of the motherboard (largest socket highlighted below in red) and do the same with the mouse ribbon (socket highlighted below in red on middle of board). Taking note of which way around they were. Then unplug the cable that transfers signal between the 4 top buttons above the keyboard from the motherboard (socket highlighted below in red, top left of the board).

Top of Eee PC 901 Mother board

Now the top piece of the plastic chassis needs to remain intact with the plastic hinge covers which are also part of the plastic running along the top of the top four silver buttons. this can be seen two images above. So this means quite a struggle to prise the plastic parts apart. Take your time. Lots of wiggling rather than excessive force is recommended.
Now once you have the top half of the plastic chassis separate. You’ll need to unplug the CPU fan wire (socket highlighted above in green). Then remove the two screws holding the motherboard to the base part of the plastic chassis. These are by the hinges with white arrows on the motherboard pointing to them.
Now there’s another cable at the bottom edge of the motherboard that needs to be unplugged. This can be seen at the bottom of the image below in the green highlight box.

Bottom of Eee PC 901 Mother board

Now wiggle the motherboard up left side first with the audio ports rather than the right side with the VGA port as the VGA port is sticking through the plastic chassis further.
Now you can finally remove the 4GB mini PCIe SATA SSD (socket shown above in blue highlight box) and replace with a larger faster one.
I used a 64GB A-Data SX300 mSATA SSD.

See my next post for Reassembly of the Eee PC 901

Interweb content I used which aided my decisions and the process:

http://forum.eeeuser.com/index.php?/topic/76496-eee-pc-901-how-do-i-upgrade-my-ssd-drives/
http://forums.tweaktown.com/runcore/38715-runcore-50mm-pro-sata-pci-e-ssd-901-a.html
Directions with images on how to pull the 901 apart but a little ambiguous and missing essential info
Pictures and speed test

In a subsequent post I’ll go through Setting up Kali Linux on the Eee PC 901

Data Centre in a Rack

June 9, 2012

I recently took the plunge to install some of my more used networking components into a server rack.
I’d been putting this off for a few years.
Most of these components have been projects of mine which I’ve already blogged on in various places on this blog.
The obvious places are the following

There are also many other topics I’ve blogged on that form part of the work gone into these components and set-up of.
Check them out.

There’s also a home made router in an old $30 desktop pc run from a CF card.

Small Data Centre

Home Rack Server

HPR Pod cast on a bunch of good tools useful for setting up and maintaining an Open Source Data Centre.
ep0366 :: The Open Source Data Center

Questions welcome.
I’m happy to provide directions and insights from my experience.

Quick walk through, of my UPS library

August 4, 2011

Part three of a three part series

On setting up a UPS solution, to enable clean shutdown of vital network components.

In this post, we’ll be reviewing the library that performs the shutting down of our servers.

When I started on the library PowerOffUPSGuests.dll,
my thoughts were, if I’m going to do this, I wanted it to be extensible.
Able to shutdown pretty much any machines, requiring a clean shutdown due to power failure.
What I’ve done is left points to be easily extended in the future, when new requirements present themselves.

Source Code

The PowerOffUPSGuests repository is on bitbucket

I’m assuming you know how to use Mercurial and have it installed on your dev machine.
If you’re not familiar with Mercurial (hg) There’s a little here to get your feet wet.
Besides that, there is plenty of very good documentation on the net.
For starters, you’ll need to create a directory that you want to have as your repository.
For example, I use C:\Scripts.
Set this directory as a repository.
Then from within the directory,
issue an hg pull https://bitbucket.org/LethalDuck/poweroffupsguests
Then update your working directory to the tip of the local repository.

You’ll need a BinaryMist.PowerOffUPSGuests.dll.config in your <repository>\UPS\PowerOffUPSGuests\PowerOffUPSGuests\
which should look something like the following.
At this stage I’ve only been shutting down an ESXi host.
Replace the value at line 22 with the user that has privileges to perform shutdown on the server.
Replace the value at line 23 with the absolute path to the password file you’re about to generate.
Line 28 is the class name of the ServerController.
Line 34 denotes whether or not the Initiator will perform the shutdowns synchronously or asynchronously
The Server[n] and ServerPort[n] values that are commented out, are used when you want to intercept the messages being sent to/from the target server.
This is useful for examination and to help build the appropriate messages that the target server expects.

<?xml version="1.0"?>
<configuration>
   <assemblySettings>
      <!--As aditional target servers are added to the queue to be shutdown
         Keep the same nameing convention used below
         Just increment the suffix number for each target servers key.
         The first target suffix must start at 0.
         Additional target suffix's must be sequential.
         -->

      <!--Target servers to be shutdown-->

      <!--FreeNAS-->
      <!--add key="ServerUser0" value="YourUser"/>
      <add key="ServerUserPwFile0" value="Absolute directory that your password file resides\FileServerPw"/-->
      <!--add key="Server0" value="127.0.0.1"/--><!--localhost used for interception-->
      <!--add key="Server0" value="YourFileServerName"/-->
      <!--add key="ServerPort0" value="8080"/--><!--port used for interception-->
      <!--add key="ServerPort0" value="443"/-->
      <!--add key="Controller0" value="FreeNASController"/-->

      <!--ESXi-->
      <add key="ServerUser0" value="YourUser"/>
      <add key="ServerUserPwFile0" value="Absolute directory that your password file resides\VMHostPw"/>
      <!--add key="Server1" value="127.0.0.1"/--><!--localhost used for interception-->
      <add key="Server0" value="YourVSphereHostName"/>
      <!--add key="ServerPort1" value="8080"/--><!--port used for interception-->
      <add key="ServerPort0" value="443"/>
      <add key="Controller0" value="VMServerController"/>

      <!--Assembly settings-->

      <add key="LogFilePath" value="Some absolute path\Log.txt"/>
      <add key="CredentialEntropy" value="A set of comma seperated digits"/><!--"4,2,7,9,1" for example-->
      <add key="Synchronicity" value="Synchronous"/> <!--Check other values in Initiator.Synchronicity-->
      <add key="IgnoreSslErrors" value="true"/>
      <add key="Debug" value="true"/>

   </assemblySettings>
<startup><supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.0"/></startup></configuration>

Build the solution

You’ll now need to navigate to <repository>\UPS\PowerOffUPSGuests\
and run the file PowerOffUPSGuests.sln.
Build.
You should now notice a couple of binaries in <repository>\UPS\
along with the libraries config file.
As I mentioned in part two, the PCNS will execute <repository>\UPS\PowerOff.bat which will run PowerOffUPSGuests.ps1.
Which will inturn kick off the BinaryMist.PowerOffUPSGuests.dll which does the work.

Generate the encrypted password file

Just run the BinaryMist.PasswordFileCreator.exe.
This will provide the required user prompts to capture the password for the vSphere host you’re intending to shutdown.
Or if you would like to extend the project and create a specialized ServerController for your needs.
You can use the BinaryMist.PasswordFileCreator to capture any credentials and save to file.

The code that performs the capture and encryption looks like the following:


        static void Main() {
            CreatePasswordFile();
        }

        /// <summary>
        /// Provides interactive capture for the insertion of an encrypted password,
        /// based on the ServerUserPwFile0 specified in the BinaryMist.PowerOffUPSGuests.dll.config file.
        /// </summary>
        public static void CreatePasswordFile() {

            bool validPath;
            string path = null;
            string RetryMessage = "Please try again.";

            Console.WriteLine("You must create the password file running under" + Initiator.NewLine + "the same account that will run BinaryMist.PowerOffUPSGuests.");

            do {
                Console.WriteLine(
                    "From the BinaryMist.PowerOffUPSGuests.dll.config file." + Initiator.NewLine +
                    "Please specify the ServerUserPwFile[n] value" + Initiator.NewLine +
                    "for the encrypted Password to be stored to." + Initiator.NewLine +
                    "This must be a valid path" + Initiator.NewLine
                );

                try {
                    validPath = true;
                    path = Path.GetFullPath(Console.ReadLine());

                    if (!
                        ((IEnumerable<string>)ConfigReader.Read.AllKeyVals.Values)
                        .Contains<string>(
                            path, StringComparer.CurrentCultureIgnoreCase
                        )
                    ) {
                        Console.WriteLine(Initiator.NewLine);
                        Console.WriteLine("The value that was entered" + Initiator.NewLine +
                            "was not one of the specified values for ServerUserPwFile[n]");
                        Console.WriteLine(RetryMessage + Initiator.NewLine);
                        validPath = false;
                    }
                } catch (Exception) {
                    Console.WriteLine(Initiator.NewLine + "An invalid path was entered." + Initiator.NewLine + RetryMessage + Initiator.NewLine);
                    validPath = false;
                }
            } while (validPath == false);

            Console.WriteLine(
                Initiator.NewLine
                + "The password you are about to enter"
                + Initiator.NewLine
                + "will be encrypted to file \"{0}\""
                , path
            );

            byte[] encryptedBytes = ProtectedData.Protect(
                new ASCIIEncoding().GetBytes(pWord),
                ServerAdminDetails.CredentialEntropy(),
                DataProtectionScope.CurrentUser
            );
            File.WriteAllBytes(path, encryptedBytes);

            Console.WriteLine(
                Initiator.NewLine
                + Initiator.NewLine
                + string.Format(
                    "The password you just entered has been encrypted"
                    + Initiator.NewLine
                    + "and saved to {0}"
                    , path
                )
            );
            Console.WriteLine(Initiator.NewLine + "Press any key to exit");
            Console.ReadKey(true);
        }

 


        private static string pWord {
            get {
                bool passWordsMatch = false;
                bool firstAttempt = true;
                string passWord = null;
                while (!passWordsMatch) {

                    if (!firstAttempt)
                        Console.WriteLine(Initiator.NewLine + "The passwords did not match." + Initiator.NewLine);

                    Console.WriteLine(Initiator.NewLine + "Please enter the password..." + Initiator.NewLine);
                    string passWordFirstAttempt = GetPWordFromUser();
                    Console.WriteLine(Initiator.NewLine + Initiator.NewLine + "Please confirm by entering the password once again..." + Initiator.NewLine);
                    string passWordSecondAttempt = GetPWordFromUser();

                    if (string.Compare(passWordFirstAttempt, passWordSecondAttempt) == 0) {
                        passWordsMatch = true;
                        passWord = passWordFirstAttempt;
                    }
                    firstAttempt = false;
                }
                Console.WriteLine(Initiator.NewLine + Initiator.NewLine + "Success, the passwords match." + Initiator.NewLine);
                return passWord;
            }
        }

 


        private static string GetPWordFromUser() {
            string passWord = string.Empty;
            ConsoleKeyInfo info = Console.ReadKey(true);
            while (info.Key != ConsoleKey.Enter) {
                if (info.Key != ConsoleKey.Backspace) {
                    if (info.KeyChar < 0x20 || info.KeyChar > 0x7E) {
                        info = Console.ReadKey(true);
                        continue;
                    }
                    passWord += info.KeyChar;
                    Console.Write("*");
                    info = Console.ReadKey(true);
                } else if (info.Key == ConsoleKey.Backspace) {
                    if (!string.IsNullOrEmpty(passWord)) {
                        passWord = passWord.Substring
                            (0, passWord.Length - 1);
                        Console.SetCursorPosition(Console.CursorLeft - 1, Console.CursorTop);
                        Console.Write(' ');
                        Console.SetCursorPosition(Console.CursorLeft - 1, Console.CursorTop);
                    }
                    info = Console.ReadKey(true);
                }
            }
            return passWord;
        }

Once you’ve created the password file, you’re pretty much ready to start testing.
If you’ve followed the directions covered in the first two parts of this series, you should be good to go.
Part one.
Part two.

We’ll quickly go through some of the more interesting parts of the code…

The ConfigReader

The constructor loads the _settings IDictionary by calling the ReadConfig function.
.net libraries don’t usually contain an assembly config file.
This is how we get around it.
We read the name of the assemblies config file (see line 70).
We then load the configuration into an XmlDocument.
Create and populate the XmlNodeList.
Return the populated IDictionary.

    /// <summary>
    /// Reads the libraries configuration into memory.
    /// Provides convienient readonly access of the configuration from a single instance.
    /// </summary>
    public class ConfigReader {

        #region singleton initialization
        private static readonly Lazy<ConfigReader> _instance = new Lazy<ConfigReader>(() => new ConfigReader());

        /// <summary>
        /// constructor that sets the value of our "_settings" variable
        /// </summary>
        private ConfigReader() {
            _settings = ReadConfig(Assembly.GetCallingAssembly());
        }

        /// <summary>
        /// The first time Read is called must be from within this assembly,
        /// in order to create the instance of this class for the containing assembly.
        /// </summary>
        public static ConfigReader Read {
            get {
                return _instance.Value;
            }
        }
        #endregion

        /// <summary>
        /// settings to be used throughout the class
        /// </summary>
        private IDictionary _settings;
        /// <summary>
        /// constant name for the node name we're looking for
        /// </summary>
        private const string NodeName = "assemblySettings";

        /// <summary>
        /// class Indexer.
        /// Provides the value of the specified key in the config file.
        /// If the key doesn't exist, an empty string is returned.
        /// </summary>
        public string this[string key] {
            get {
                string settingValue = null;

                if (_settings != null) {
                    settingValue = _settings[key] as string;
                }

                return settingValue ?? string.Empty;
            }
        }

        public IDictionary<string, string> AllKeyVals {
            get {
                IDictionary<string, string> settings = new Dictionary<string, string>();
                foreach(DictionaryEntry item in _settings) {
                    settings.Add((string)item.Key, (string)item.Value);
                }
                return settings;
            }
        }

        /// <summary>
        /// Open and parse the config file for the provided assembly
        /// </summary>
        /// <param name="assembly">The assembly that has a config file.</param>
        /// <returns></returns>
        private static IDictionary ReadConfig(Assembly assembly) {
            try {
                string cfgFile = assembly.CodeBase + ".config";

                XmlDocument doc = new XmlDocument();
                doc.Load(new XmlTextReader(cfgFile));
                XmlNodeList nodes = doc.GetElementsByTagName(NodeName);

                foreach (XmlNode node in nodes) {
                    if (node.LocalName == NodeName) {
                        DictionarySectionHandler handler = new DictionarySectionHandler();
                        return (IDictionary)handler.Create(null, null, node);
                    }
                }
            } catch (Exception e) {
                Logger.Instance.Log(e.Message);
            }

            return (null);
        }

        #region Config related settings

        /// <summary>
        /// Readonly value, specifying whether debug is set to true in the assemblySettings of the BinaryMist.PowerOffUPSGuests.dll.config file.
        /// </summary>
        public bool Debug {
            get {
                if (_debug != null)
                    return _debug == true ? true : false;
                _debug = Read["Debug"] == "true" ? true : false;
                return _debug == true ? true : false;
            }
        }
        private bool? _debug;

        #endregion

    }

The Initiator


private void ShutdownSynchronously(Queue<ServerController> serverControllers) {
    foreach (ServerController serverController in serverControllers) {
        serverController.Shutdown();
    }
}

private void ShutdownAsynchronously(Queue<ServerController> serverControllers) {
    Action[] shutdownActions = new Action[serverControllers.Count];
    ServerController[] serverControllerArray = serverControllers.ToArray();

    for (int i = 0; i < serverControllerArray.Length; i++) {
        shutdownActions[i] = serverControllerArray[i].Shutdown;
    }

    try {
        Parallel.Invoke(shutdownActions);
    }
        // No exception is expected in this example, but if one is still thrown from a task,
        // it will be wrapped in AggregateException and propagated to the main thread. See MSDN example
    catch (AggregateException e) {
        Logger.Instance.Log(string.Format("An action has thrown an exception. THIS WAS UNEXPECTED.\n{0}", e.InnerException));
        throw new Exception();
    }
}

public string InitShutdownOfServers() {
    Logger.Instance.LogTrace();
    Queue<ServerController> serverControllers = new Queue<ServerController>();

    try {
        foreach (ServerAdminDetails serverAdminDetail in ServerAdminDetails.QueuedDetails) {

            Type t = Type.GetType(GetType().Namespace + "." + serverAdminDetail.ServerControllerType);
            serverControllers.Enqueue(Activator.CreateInstance(t, serverAdminDetail) as ServerController);
        }
    } catch(Exception e) {
        Logger.Instance.Log("Exception occured while enqueueing the server controllers. Details follow:" +
            NewLine +
            e.ToString()
        );
        throw;
    }

    bool ignoreCase = true;
    Synchronicity synchronicity = (Synchronicity)Enum.Parse(typeof (Synchronicity), ConfigReader.Read["Synchronicity"], ignoreCase);

    if(synchronicity == Synchronicity.Synchronous)
        ShutdownSynchronously(serverControllers);
    else
        ShutdownAsynchronously(serverControllers);

    return "InitShutdownOfServers successfully executed.";
}

In order to place the ServerController‘s on the serverController‘s Queue (line 33)… via the iterator provided by the Queue of ServerAdminDetails returned from
the static QueuedDetails property of ServerAdminDetails,
We must first instantiate the single instance of ServerAdminDetails,
which is what line 30 does.

In order for ServerAdminDetails to be constructed, the static _queue member (which is a Lazy of Queue of ServerAdminDetails) must be initialized first.
In order for the _queue member to be initialized with a Queue of ServerAdminDetails, the static QueueServersForShutdown procedure must be called.

This is where we pull out the values from the BinaryMist.PowerOffUPSGuests.dll.config shown above with the help of the ConfigReader.
As you can see, we iterate through the config file, building up the Queue of ServerAdminDetails  until we’ve read all the appropriate values.
Each pass through the loop instantiates a new ServerAdminDetails (non singleton because of inside class scope) with the values we pulled from the config file.

The ServerAdminDetails

    /// <summary>
    /// Provides the administration details of the servers listed in the BinaryMist.PowerOffUPSGuests.dll.config file.
    /// </summary>
    public class ServerAdminDetails {

        #region singleton initialization
        private static readonly Lazy<Queue<ServerAdminDetails>> _queue = new Lazy<Queue<ServerAdminDetails>>(QueueServersForShutdown);

        private ServerAdminDetails(string serverController, string serverName, string serverPort, string userName, byte[] serverCredential) {
            ServerControllerType = serverController;
            ServerName = serverName;
            ServerPort = serverPort;
            UserName = userName;
            Password = serverCredential;
        }

        /// <summary>
        /// Provides the process wide single instance queue of each servers admin details.
        /// </summary>
        public static Queue<ServerAdminDetails> QueuedDetails {
            get {
                return _queue.Value;
            }
        }
        #endregion

        private static Queue<ServerAdminDetails> ServersQueuedForShutdown { get; set; }

        private static byte[] GetMyCredential(string pWFileName) {
            try {
                return File.ReadAllBytes(pWFileName);
            } catch(Exception e) {
            string error = string.Format(
                "Error occured while instantiating a ServerAdminDetails instance for queueing. Specificaly while reading bytes from the following file: {0}{1}Exception details follow.{1}{2}",
                pWFileName,
                Initiator.NewLine,
                    e
                );
                Logger.Instance.Log(error);
                throw new Exception(error);
            }
        }

        private static Queue<ServerAdminDetails> QueueServersForShutdown() {

            ServersQueuedForShutdown = new Queue<ServerAdminDetails>();

            const int firstServerIndex = 0;
            int serverCount = firstServerIndex;
            string empty = string.Empty;

            do
            {
                string controller = ConfigReader.Read["Controller" + serverCount];
                string server = ConfigReader.Read["Server" + serverCount];
                string serverPort = ConfigReader.Read["ServerPort" + serverCount];
                string serverUser = ConfigReader.Read["ServerUser" + serverCount];
                string serverUserPwFile = ConfigReader.Read["ServerUserPwFile" + serverCount];

                if (controller == empty || server == empty || serverPort == empty || serverUser == empty || serverUserPwFile == empty)
                    break;

                ServersQueuedForShutdown.Enqueue(
                    new ServerAdminDetails(
                        controller,
                        server,
                        serverPort,
                        serverUser,
                        GetMyCredential(Path.GetFullPath(serverUserPwFile))
                    )
                );

                Logger.Instance.Log (
                    string.Format (
                        "Server admin details of Controller: {0}, Server: {1}, ServerPort: {2}, ServerUser: {3}, ServerUserPwFile: {4} added to queued element number {5}.",
                        controller,
                        server,
                        serverPort,
                        serverUser,
                        serverUserPwFile,
                        serverCount
                    )
                );
                serverCount++;

            } while (true);
            return ServersQueuedForShutdown;
        }

        /// <summary>
        /// Retreives the entropy found in the BinaryMist.PowerOffUPSGuests.dll.config file, used to encrypt all the passwords.
        /// </summary>
        /// <returns>byte[]</returns>
        public static byte[] CredentialEntropy() {
            string[] numbers = ConfigReader.Read["CredentialEntropy"].Split(',');
            byte[] entropy = new byte[numbers.Length];
            for (int i = 0; i < numbers.Length; i++) {
                entropy[i] = Byte.Parse(numbers[i]);
            }
            return entropy;
        }

        /// <summary>
        /// The name of the ServerController child type.
        /// </summary>
        internal string ServerControllerType { get; private set; }

        /// <summary>
        /// The name of the server
        /// </summary>
        internal string ServerName { get; private set; }

        /// <summary>
        /// The port of the server
        /// </summary>
        internal string ServerPort { get; private set; }

        /// <summary>
        /// The user name for the server
        /// </summary>
        internal string UserName { get; private set; }

        /// <summary>
        /// The password for the user
        /// </summary>
        internal byte[] Password { get; private set; }
    }

Back to line 33 of the Initiator.
Now that we can access each ServerAdminDetails within the Queue of ServerAdminDetails via the Queue iterator provided by ServerAdminDetails.QueuedDetails.
We can create the ServerController child types using the late bound Activator.CreateInstance method, based on the ServerAdminDetails.ServerControllerType.
A reference to each ServerController child instance is added to the serverControllers queue.
The Shutdown procedure for each ServerController child is then called.

The ServerController

Notice the constructor which has been called by the child’s constructor, calls back to the child’s AssembleRequests procedure before completing.

    /// <summary>
    /// Controls the process of shutting down the associated server.
    /// </summary>
    /// <remarks>
    /// An instance of this class is created indirectly via the more specific concrete creators for each server that requires shutdown.
    /// Plays the part of the Creator, in the Factory Method pattern.
    /// </remarks>
    internal abstract class ServerController {

        protected static readonly string NewLine = Initiator.NewLine;

        protected enum RequestMethod {
            Get,
            Post
        }

        /// <summary>
        /// Constructor for the <see cref="ServerController"/> class.
        /// Called via the more specific children to initialize the less specific members.
        /// </summary>
        /// <param name="serverAdminDetails">
        /// The details required to create the messages that need to be sent to the server in order to perform the shutdown.</param>
        public ServerController(ServerAdminDetails serverAdminDetails) {
            ServerAdminDetails = serverAdminDetails;
            RequestAssembler = new RequestAssembler();
            SoapEnvelopes = new Queue<XmlDocument>();
            this.AssembleRequests();
        }

        protected ServerAdminDetails ServerAdminDetails { get; set; }

        protected RequestAssembler RequestAssembler { get; set; }

        /// <summary>
        /// Reference a <see cref="System.Collections.Generic.Queue{System.Xml.XmlDocument}">queue</see> of soap envelopes.
        /// used by the children of this class to send to the server.
        /// </summary>
        public Queue<XmlDocument> SoapEnvelopes { get; protected set; }

        /// <summary>
        /// Initial preparation of messages that will be sent to the server to perform shutdown.
        /// </summary>
        /// <remarks>Factory method.</remarks>
        public abstract void AssembleRequests();

        /// <summary>
        /// Completes the compilation of the sequence of messages that need to be sent to the server in order to perform the shutdown.
        /// Sends the messages.
        /// </summary>
        public abstract void Shutdown();

        protected void NotifyOfShutdown() {
            Logger.Instance.Log(
                string.Format(
                    "{0}.Shutdown on server: {1} has now been executed.",
                    ServerAdminDetails.ServerControllerType,
                    ServerAdminDetails.ServerName
                )
            );
        }

        protected bool SimplePing() {
            string serverName = ServerAdminDetails.ServerName;
            Logger.Instance.Log(string.Format("Performing Ping test on server: {0}", serverName));
            Ping pingSender = new Ping();
            int pingRetry = 3;
            PingReply reply = null;
            for (int i = 0; i < pingRetry; i++) {
                try {
                    //may take a couple of tries, as may time out due to arp delay
                    pingSender.Send(serverName);
                    Logger.Instance.Log(string.Format("Initiating Ping number {0} of {1} on server: {2}. ", i + 1, pingRetry, serverName));
                    reply = pingSender.Send(serverName);

                    if (reply.Status == IPStatus.Success) break;

                } catch (Exception e) {
                    Logger.Instance.Log(e.ToString());
                }
            }

            bool optionsAvailable = reply.Options != null;
            string noOptions = "No Options available";

            Logger.Instance.Log(
                "Reply status for server: " + serverName + " was " + reply.Status + ". " + NewLine +
                "Address: " + reply.Address.ToString() + ". " + NewLine +
                "RoundTrip time: " + reply.RoundtripTime + ". " + NewLine +
                "Time to live: " + ((optionsAvailable) ? reply.Options.Ttl.ToString() : noOptions) + ". " + NewLine +
                "Don't fragment: " + ((optionsAvailable) ? reply.Options.DontFragment.ToString() : noOptions) + ". " + NewLine +
                "Buffer size: " + reply.Buffer.Length + ". "
                );

            return reply.Status == IPStatus.Success;
        }
    }

The ServerController children

The AssembleRequests constructs as much of the SOAP envelopes as it can,
without knowing all the information that the target server will provide to be able to complete the messages before being sent.
The RequestAssembler‘s CreateSoapEnvelope does the assembly of the SOAP envelope.
You’ll notice that the RequestAssembler‘s CreateLoginSoapEnvelope takes an extra argument.
The ServerAdminDetails is passed so that the target server’s credentials can be included in the SOAP envelope.
The SOAP envelopes are then queued ready for dispatch.

Now when line 02 or 11 of the Initiator is executed,
line 143 of the VMServerController will be called. That’s Shutdown.
Then we dequeue, send request, and process the response in DequeueSendRequestProcessResponse.
Passing in an optional parameter of Action of HttpWebResponse (the lambda).
From DequeueSendRequestProcessResponse,
we dequeue each SOAP envelope and call CreateWebRequest,
which in-turn, palms the work it knows about to the less specific RequestAssembler‘s CreateWebRequest as a lambda.
Now in DequeueSendRequestProcessResponse, when we get our initialized HttpWebRequest back,
We pass both the SOAP envelope and the HttpWebRequest to the RequestAssembler‘s InsertSoapEnvelopeIntoWebRequest‘s procedure to do the honors.

    /// <summary>
    /// Controls the process of shutting down the associated vSphere server.
    /// </summary>
    /// <remarks>
    /// Plays the part of the Concrete Creator, in the Factory Method pattern.
    /// </remarks>
    internal class VMServerController : ServerController {

        private static string _operationIDEndTag = "</operationID>";
        private static string _operationIDTags = "<operationID>" + _operationIDEndTag;
        private uint _operationIDaVal = 0xAC1CF80C;
        private uint _operationIDbVal = 0x00000000;
        private const int EstimatedHeaderSize = 45;

        private enum RequestType {
            Hello, HandShake, Login, Shutdown
        }

        private string _host;
        private string _uRL;
        private readonly string _action = @"""urn:internalvim25/4.1""";
        private readonly string _userAgent = @"VMware VI Client/4.0.0";
        private KeyValuePair<string, string> _cookie;

        public VMServerController(ServerAdminDetails serverAdminDetails) : base(serverAdminDetails) {

        }

        /// <summary>
        /// Loads the <see cref="ServerController.SoapEnvelopes">SoapEnvelopes</see> with the sequence of messages
        /// that need to be sent to the server in order to perform the shutdown.
        /// </summary>
        /// <remarks>
        /// Factory method implementation
        /// </remarks>
        public override void AssembleRequests() {
            Logger.Instance.LogTrace();
            _host = "https://" + ServerAdminDetails.ServerName + ":" +ServerAdminDetails.ServerPort;
            _uRL = "/sdk";
            SoapEnvelopes.Enqueue(CreateHelloEnvelope());
            SoapEnvelopes.Enqueue(CreateHandshakeEnvelope());
            SoapEnvelopes.Enqueue(CreateLoginEnvelope());
            SoapEnvelopes.Enqueue(CreateShutdownEnvelope());
        }

        private string InitialHeaderContent() {
            StringBuilder headerContent = new StringBuilder(_operationIDTags, EstimatedHeaderSize);
            headerContent.Insert(headerContent.ToString().IndexOf(_operationIDEndTag), _operationIDaVal.ToString("X8") + "-" + (++_operationIDbVal).ToString("X8"));
            return headerContent.ToString();
        }

        private XmlDocument CreateHelloEnvelope() {
            Logger.Instance.LogTrace();
            string bodyContent = @"
    <RetrieveServiceContent xmlns=""urn:internalvim25"">
      <_this xsi:type=""ManagedObjectReference"" type=""ServiceInstance"" serverGuid="""">ServiceInstance</_this>
    </RetrieveServiceContent>";
            return RequestAssembler.CreateSoapEnvelope(InitialHeaderContent(), bodyContent);
        }

        private XmlDocument CreateHandshakeEnvelope() {
            Logger.Instance.LogTrace();
            string bodyContent = @"
    <RetrieveInternalContent xmlns=""urn:internalvim25"">
      <_this xsi:type=""ManagedObjectReference"" type=""ServiceInstance"" serverGuid="""">ServiceInstance</_this>
    </RetrieveInternalContent>";
            return RequestAssembler.CreateSoapEnvelope(InitialHeaderContent(), bodyContent);
        }

        private XmlDocument CreateLoginEnvelope() {
            Logger.Instance.LogTrace();
            string bodyContent = @"
    <Login xmlns=""urn:internalvim25"">
      <_this xsi:type=""ManagedObjectReference"" type=""SessionManager"" serverGuid="""">ha-sessionmgr</_this>
      <userName></userName>
      <password></password>
      <locale>en_US</locale>
    </Login>";
            try {
                // As VMware insist on putting credentials in the SOAP body what else can we do?
                return RequestAssembler.CreateLoginSoapEnvelope(InitialHeaderContent(), bodyContent, ServerAdminDetails);
            } catch(InvalidCredentialException e) {
                string error = string.Format(
                    "Error occured during a call to CreateLoginSoapEnvelope, for server: {0}.{1}Exception details follow.{1}{2}",
                    ServerAdminDetails.ServerName,
                    NewLine,
                    e
                );
                Logger.Instance.Log(error);
                throw new Exception(error);
            }
        }

        private XmlDocument CreateShutdownEnvelope() {
            Logger.Instance.LogTrace();
            string bodyContent = @"
    <ShutdownHost_Task xmlns=""urn:internalvim25"">
      <_this xsi:type=""ManagedObjectReference"" type=""HostSystem"" serverGuid="""">ha-host</_this>
      <force>true</force>
    </ShutdownHost_Task>";
            return RequestAssembler.CreateSoapEnvelope(InitialHeaderContent(), bodyContent);
        }

        private HttpWebRequest CreateWebRequest(string uRL, KeyValuePair<string, string> cookie) {
            return RequestAssembler.CreateWebRequest(
                uRL,
                (request)=>{
                    request.Method = RequestMethod.Post.ToString();
                    request.UserAgent = _userAgent;
                    request.ContentType = "text/xml; charset=\"utf-8\"";
                    request.Headers.Add("SOAPAction", _action);
                    request.Accept = "text/xml";
                    request.KeepAlive = true;

                    if (!string.IsNullOrEmpty(cookie.Key))
                        request.Headers.Add("Cookie", cookie.Key + "=" + cookie.Value);
                }
            );
        }

        private void DequeueSendRequestProcessResponse(RequestType requestType, KeyValuePair<string, string> cookie, Action<HttpWebResponse> additionalResponseProcessing = null) {
            Logger.Instance.Log(string.Format("Will now attempt sending {0} message to server: {1}", requestType, ServerAdminDetails.ServerName));
            XmlDocument soapEnvelope = SoapEnvelopes.Dequeue();
            HttpWebRequest httpWebRequest = CreateWebRequest(_host + _uRL, cookie);
            RequestAssembler.InsertSoapEnvelopeIntoWebRequest(soapEnvelope, httpWebRequest);

            string soapResult;
            using (HttpWebResponse response = (HttpWebResponse)httpWebRequest.GetResponse())
            using (Stream responseStream = response.GetResponseStream())
            using (StreamReader streamReader = new StreamReader(responseStream)) {
                //pull out the bits we need for the next request.

                if (additionalResponseProcessing != null) {
                    additionalResponseProcessing(response);
                }

                soapResult = streamReader.ReadToEnd();
            }
        }

        /// <summary>
        /// Perform the shutdown of the server specified in the <see cref="ServerAdminDetails">server admin details</see>.
        /// </summary>
        public override void Shutdown() {

            bool serverOnline = SimplePing();
            Logger.Instance.Log(
                serverOnline
                    ? string.Format("Initiating sending of {0} to server: {1}", RequestType.Hello, ServerAdminDetails.ServerName)
                    : string.Format("Could not reach server: {0}. Aborting shutdown of server: {0}", ServerAdminDetails.ServerName)
            );

            ServicePointManager.ServerCertificateValidationCallback += ValidateRemoteCertificate;

            KeyValuePair<string, string> emptyCookie = new KeyValuePair<string, string>();
            DequeueSendRequestProcessResponse(
                RequestType.Hello,
                emptyCookie,
                (response)=> {
                    string[] setCookieElementsResponse = response.Headers["Set-Cookie"].Split(new[] { "\"" }, StringSplitOptions.RemoveEmptyEntries);
                    _cookie = new KeyValuePair<string, string>(setCookieElementsResponse[0].TrimEnd('='), "\"" + setCookieElementsResponse[1] + "\"");
                }
            );
            DequeueSendRequestProcessResponse(RequestType.HandShake, _cookie);
            DequeueSendRequestProcessResponse(RequestType.Login, _cookie);
            DequeueSendRequestProcessResponse(RequestType.Shutdown, _cookie);
            NotifyOfShutdown();
        }

        private static string RemoteCertificateDetails(X509Certificate certificate) {
            return string.Format(
                "Details of the certificate provided by the remote party are as follows:" + NewLine +
                "Subject: {0}" + NewLine +
                "Issuer: {1}",
                certificate.Subject,
                certificate.Issuer
            );
        }
    }

The RequestAssembler

    /// <summary>
    /// Provides ancillary operations for <see cref="ServerController">server controllers</see>
    /// that assist in the creation of the requests intended for dispatch to the servers requiring shutdown.
    /// </summary>
    internal class RequestAssembler {

        private static string _soapEnvelope = @"<soap:Envelope xmlns:xsd='http://www.w3.org/2001/XMLSchema' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns:soap='http://schemas.xmlsoap.org/soap/envelope/'>
    <soap:Header>
    </soap:Header>
    <soap:Body>
    </soap:Body>
</soap:Envelope>";

        /// <summary>
        /// Produces a SOAP envelope in XML form with the server admins user credentials
        /// </summary>
        /// <param name="headerContent">Content targeted for between the header tags.</param>
        /// <param name="bodyContent">Content targeted for between the body tags.</param>
        /// <param name="serverAdminDetails">The <see cref="ServerAdminDetails"/> instance used to access the user name and password.</param>
        /// <returns>The SOAP envelope.</returns>
        public XmlDocument CreateLoginSoapEnvelope(string headerContent, string bodyContent, ServerAdminDetails serverAdminDetails) {

            string failMessage = "The credentials were not correctly set. Username: {0} Password byte count: {1}";
            if (string.IsNullOrEmpty(serverAdminDetails.UserName) || serverAdminDetails.Password.Length < 1)
                throw new InvalidCredentialException(string.Format(failMessage, serverAdminDetails.UserName, serverAdminDetails.Password.Length));

            StringBuilder bodyContentBuilder= new StringBuilder(bodyContent, bodyContent.Length + 10);
            bodyContentBuilder.Insert(
                bodyContentBuilder.ToString().IndexOf("</userName>"),
                serverAdminDetails.UserName
            );

            bodyContentBuilder.Insert(
                bodyContentBuilder.ToString().IndexOf("</password>"),
                decryptedCredential(serverAdminDetails)
            );

            return CreateSoapEnvelope(headerContent, bodyContentBuilder.ToString());
        }

        private string decryptedCredential(ServerAdminDetails serverAdminDetails) {
            try {
                return new ASCIIEncoding().GetString(
                    ProtectedData.Unprotect(
                        serverAdminDetails.Password,
                        ServerAdminDetails.CredentialEntropy(),
                        DataProtectionScope.CurrentUser
                    )
                );
            } catch(CryptographicException e) {

                // Retrieve the exception that caused the current
                // CryptographicException exception.
                Exception innerException = e.InnerException;
                string innerExceptionMessage = "";
                if (innerException != null) {
                    innerExceptionMessage = innerException.ToString();
                }

                // Retrieve the message that describes the exception.
                string message = e.Message;

                // Retrieve the name of the application that caused the exception.
                string exceptionSource = e.Source;

                // Retrieve the call stack at the time the exception occured.
                string stackTrace = e.StackTrace;

                // Retrieve the method that threw the exception.
                System.Reflection.MethodBase targetSite = e.TargetSite;
                string siteName = targetSite.Name;

                // Retrieve the entire exception as a single string.
                string entireException = e.ToString();

                // Get the root exception that caused the current
                // CryptographicException exception.
                Exception baseException = e.GetBaseException();
                string baseExceptionMessage = "";
                if (baseException != null) {
                    baseExceptionMessage = baseException.Message;
                }

                Logger.Instance.Log(
                    "Caught an unexpected exception:" + Initiator.NewLine
                    + entireException + Initiator.NewLine
                    + Initiator.NewLine
                    + "Properties of the exception are as follows:" + Initiator.NewLine
                    + "Message: " + message + Initiator.NewLine
                    + "Source: " + exceptionSource + Initiator.NewLine
                    + "Stack trace: " + stackTrace + Initiator.NewLine
                    + "Target site's name: " + siteName + Initiator.NewLine
                    + "Base exception message: " + baseExceptionMessage + Initiator.NewLine
                    + "Inner exception message: " + innerExceptionMessage + Initiator.NewLine
                );
                throw;
            }
        }

        /// <summary>
        /// Produces a SOAP envelope in XML form.
        /// </summary>
        /// <param name="headerContent">Content targeted for between the header tags.</param>
        /// <param name="bodyContent">Content targeted for between the body tags.</param>
        /// <returns><see cref="System.Xml.XmlDocument">The SOAP envelope</see>.</returns>
        public XmlDocument CreateSoapEnvelope(string headerContent, string bodyContent) {

            StringBuilder sb = new StringBuilder(_soapEnvelope);

            try {
                sb.Insert(sb.ToString().IndexOf(Initiator.NewLine + "    " + "</soap:Header>"), headerContent);
                sb.Insert(sb.ToString().IndexOf(Initiator.NewLine + "    " + "</soap:Body>"), bodyContent);
            } catch(Exception e) {
                Logger.Instance.Log(e.ToString());
                throw;
            }

            XmlDocument soapEnvelopeXml = new XmlDocument();
            soapEnvelopeXml.LoadXml(sb.ToString());

            return soapEnvelopeXml;
        }

        /// <summary>
        /// Creates a web request based on the url passed in.
        /// </summary>
        /// <param name="url">The target URL of the server that will be shutdown.</param>
        /// <param name="additionalWebRequestManipulation">delegate of type
        /// <see cref="System.Action{HttpWebRequest}">Action{HttpWebRequest}</see>.
        /// This is used to take additional tasking defined in the calling procedure.
        /// This procedure creates the web request,
        /// and passes it into this parameter for the additional initialization work to be performed on the web request.
        /// </param>
        /// <returns>
        /// An initialized <see cref="System.Net.HttpWebRequest">web request</see>,
        /// ready to have a <see cref="System.Xml.XmlDocument">soap envelope</see> inserted.</returns>
        public HttpWebRequest CreateWebRequest(string url, Action<HttpWebRequest> additionalWebRequestManipulation = null) {
            HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(url);

            if(additionalWebRequestManipulation != null) {
                additionalWebRequestManipulation(webRequest);
            }
            return webRequest;
        }

        /// <summary>
        /// Insert the <see cref="System.Xml.XmlDocument">soap envelope</see> into the <see cref="System.Net.HttpWebRequest">web request</see>.
        /// </summary>
        /// <param name="soapEnvelopeXml">
        /// The <see cref="System.Xml.XmlDocument">soap envelope</see> to be inserted into the
        /// <see cref="System.Net.HttpWebRequest">web request</see>.
        /// </param>
        /// <param name="webRequest">The <see cref="System.Net.HttpWebRequest">web request</see> that the
        /// <see cref="System.Xml.XmlDocument"/>soap envelope</see> is inserted into.
        /// </param>
        public void InsertSoapEnvelopeIntoWebRequest(XmlDocument soapEnvelopeXml, HttpWebRequest webRequest) {
            using (Stream stream = webRequest.GetRequestStream()) {
                soapEnvelopeXml.Save(stream);
            }
        }

        private static void InsertByteArrayIntoWebRequest(byte[] postData, HttpWebRequest webRequest) {
            using (Stream stream = webRequest.GetRequestStream()) {
                stream.Write(postData, 0, postData.Length);
            }
            webRequest.ContentLength = postData.Length;
        }

        /// <summary>
        /// Inserts the credentials from the <see cref="ServerAdminDetails">server admmin details</see> into the
        /// <see cref="System.Net.HttpWebRequest">web request</see> supplied.
        /// </summary>
        /// <param name="serverAdminDetails">
        /// The <see cref="ServerAdminDetails">server admin details</see>
        /// containing the information for the server that the request will be sent to.
        /// </param>
        /// <param name="webRequest">
        /// The <see cref="System.Net.HttpWebRequest">web request</see> that will have the server administration credentials inserted.
        /// </param>
        public void InsertCredentialsIntoWebRequest(ServerAdminDetails serverAdminDetails, HttpWebRequest webRequest) {
            InsertByteArrayIntoWebRequest(
                new ASCIIEncoding().GetBytes("username=" + serverAdminDetails.UserName + "&password=" + decryptedCredential(serverAdminDetails)),
                webRequest
             );
        }
    }


Let me know if any of this is unclear, and requires additional explanation.

Once again, the full source code can be found here.

Preparing APC Smart-UPS 1500 clients

July 26, 2011

Part two of a three part series

on Setting up a UPS solution, to enable clean shutdown of vital network components.

This post is about setting up the software that will be responsible for cleanly shutting down servers and workstations.

We have to decide which machine/s is/are going to be used to launch our script (which in turn is run by what APC call a command file).

Currently I’ve got an old laptop I pulled out of the rubbish about 5 years ago, with Windows XP running on it.
It’s got just enough battery capacity to stay alive for long enough to receive the event from the NMC (Network Management Card) and run my .dll that issues the shutdown sequence.
A couple of EeePC 901’s have also recently been made redundant, and I may use one of those with Windows 7 installed at some stage.
Currently all of my workstations and servers that don’t have batteries, I.E. notebooks are VM’s running on ESXi.
Oh… or servers that have their entire file system loaded into volatile memory, so if they are powered off, I.E. cold shutdown, there is no possible corruption of the file system.
What you can also do is host the PCNS (PowerChute Network Shutdown) on a VM, because once the shutdown of ESXi has been initiated, there is no stopping the sequence, and the VM’s will all be cleanly shutdown.
Or better still, use more than one machine to host PCNS, as they will operate on a first in first served basis.
As you’ll see here the NMC’s firmware and PCNS are quite extensible.
The above document is recommended reading if your planning on setting up an APC UPS and want to automate clean shutdowns.
Without reading, the comms can get a little confusing.

Setting up PCNS

Install the PowerChute Network Shutdown service

You can get a copy of v2.2.3 here
I later found out that there were later versions:
v2.2.4 linked to from here, which has additional documentation.
v3.0.0 linked to from here, which has additional documentation.
Both of which were linked to from here, which has additional manuals etc.
You can find the installation guide here.
The PCNS service needs to be run as a local Administrator as the default Local System account doesn’t have sufficient rights.
In saying all that, William Tournas from APC recommended I use PCNS 2.2.1 for Windows XP.
Additional 2.2.1 resources are found here.

If using vMA with PCNS 3.0, you go through a Web UI configuration wizard once installed.
If using PCNS with Windows, the configuration is part of the install.

Either way, the steps will look similar to the following:

netstat -a

Should show that PCNS is listening on TCP and UDP ports 3052

If it’s not, you’ll need to open those ports on your firewall.

If you’re looking at using a Linux based VM to host PCNS,
VMware provide vMA (vSphere Management Assistant) a CentOS VM image.
You can get the binary here.
You’ll also need PCNS.
Take your pick of the following binaries:
2.2.4
3.0.0
Along with the documentation:
PowerChute_NetworkShutdownv2.4-ReleaseNotes.htm
PowerChute_NetworkShutdownv3.0-ReleaseNotes.htm
You’ll have to have the same ports open, as PCNS will be listening on them.
A listing of iptables for the filter (default unless otherwise specified) table should look like the following:

For an easier to read output, try the following:

sudo iptables  -L -v -n --line-numbers | column -t

Once again, if these ports are not open, you’ll have to find which script is being used to set up the rules.
I’m not sure about CentOS, but in a Debian based system, you would normally put the firewall init script in /etc/init.d/
This script would call a script that sets up the rules and one that tears them down.
I’m going to be making a post about how I set up my firewall (iptables) rules for the netfilter module on our notebooks at some stage soon.
If I haven’t already done this and you need more help, just sing out.

I found the following links quite helpful with the setup:

Link1

ESXi.pdf linked to from here linked to from here.

This has a list of the ports that are supposed to be open on pcns 2.2.3 with ESX 3.5
I think this also applies to PCNS 3.0 and ESXi 4.1 which I tried out.

Also be aware that there’s a known issue with special characters in the credentials for PCNS 3.0

I read somewhere that the PCNS needs to have the same credentials as the NMC, so just be aware of this.

Could be useful for trouble shooting vMA (vSphere Management Assistant)
I made a couple of posts there.

PowerChute Network Shutdown v3.0 – Release Notes
goes through a whole lot of issues and work-arounds with PowerChute.
For example, discuss’s the correct way to run the command file, PowerOff.bat in our case.

The APC PCNS receives an event from the AP9606 (that’s the NMC (Network Management Card)) fitted to the UPS.
The script is launched by APC PCNS from a Windows or Linux box.
I read that PCNS will always shutdown the windows machine it’s running on.
This is not true.

My attempt at using a PowerShell script utilizing mostly VMware’s cmdlets to shutdown ESXi

PowerChute has an option to ‘run this command’ but it’s limited to 8.3 paths and won’t accept command line parameters.
A separate batch file is needed (I called it poweroff.bat)
that runs the shutdown script with the parameters – but that could shut down other ESXi boxes as well if required.

I was keen to use PowerShell to perform the shutdowns, as I’d read it was quite capable and also VMware supplied a large set of management cmdlets.

Install PowerCLI from here.
read the installation guide.
As an admin, run the following:

set-executionpolicy remotesigned

Details of executionpolicy here.

If running PowerOffUPSGuests.ps1 from command shell rather than from a batch file.
You need to add the PowerCLI snapin.

PS C:\scripts&gt; Add-PSSnapin VMware.VimAutomation.Core
PS C:\scripts&gt; . .\PowerOffUPSGuests.ps1 MyESXiHostName AdminUserName

This will establish the SSL connection to MyESXiHostName

Following are the PowerShell scripts I used.

First we had to create our password file to use to log in to vSphere.
See this post for how this was done.

PowerOffUPSGuests.bat (the command file)

echo off
REM VMware would have used the Export-Console cmdlet to export the name of the PowerShell snap-in PowerCLI uses.
REM to the PowerShell console file (.psc1)

REM Invoke the command with the call operator (The ampersand).
PowerShell.exe -PSConsoleFile "C:\Program Files\VMware\Infrastructure\vSphere PowerCLI\vim.psc1" "& "C:\Scripts\PowerOffUPSGuests.ps1" MyESXiServer.MyDomain MyUser

PowerOffUPSGuests.ps1 (the script that was going to do the work)

param ( [parameter(Mandatory=$true)][string] $vSphereServername,
   [parameter(Mandatory=$true)][string] $user
)

$HostCredential = C:\Scripts\Get-myCredential.ps1 $user C:\Scripts\mp.txt

Set-StrictMode -Version 2.0
Write-Host "Establishing connection to $vSphereServername" -ForegroundColor Yellow
Connect-VIServer -Server $vSphereServername -Protocol https -Credential $HostCredential

function Stop-VMOnVMHost {
   Write-Host "Shutting down guests." -ForegroundColor Yellow

   $vM = Get-VM | Where-Object {$_.PowerState -eq "PoweredOn" -and $_.Guest.State -eq "Running"}
   Write-Host "Shutting down the following guests: $vM " -ForegroundColor Yellow
   $vM | Shutdown-VMGuest -Confirm:$False
   $seconds = 300
   Write-Host "Waiting $seconds Seconds. "
   Start-Sleep -Seconds $seconds

   $vM = Get-VM | Where-Object {$_.PowerState -eq "PoweredOn"}
   Write-Host "Stopping the following guests: $vM " -ForegroundColor Yellow
   $vM | Stop-VM -RunAsync -Confirm:$False
   $seconds = 60
   Write-Host "Waiting %seconds Seconds. "
   Start-Sleep -Seconds $seconds
}

function Stop-VMHost {
   Write-Host "Setting state of $vSphereServername to maintenance mode. " -ForegroundColor Yellow
   Get-VMHost | ForEach-Object {
      $hostName = $_.Name
      Write-Host "Putting $hostName into maintenance mode. "
      Set-VMHost -vmhost $_ -state maintenance
      Write-Host "Stopping $hostName. "
      Stop-VMHost -vmhost $_ -RunAsync
   }
}

Stop-VMOnVMHost
Stop-VMHost
Write-Host "Shutdown Complete" -ForegroundColor Yellow

Tried my script and got the following:

Shutdown-VMGuest     Operation “Shutdown VM guest.” failed for VM “MyGuestNameHere” for the following reason: fault.Restriction.summary

I had a hunch that it was due to the read only restriction I had heard about.
So tried command straight from PowerShell console…
same result.
More details here.
PowerCLI references to shutting down ESX
http://pastebin.com/HgsbSpb7
http://www.sheenaustin.com/2011/02/20/vmware-ups-shutdown-script/
http://communities.vmware.com/message/1555286
http://spininfo.homelinux.com/news/vSphere_PowerCLI/2010/06/18/Shutdown_infrastructure_on_power_outage
http://blogs.vmware.com/kb/2010/09/managing-esxi-made-easy-using-powercli.html
http://www.vmware.com/support/developer/PowerCLI/PowerCLI41U1/html/index.html

So as it turned out, VMware has removed write access from PowerCLI to ESXi, in 4.0 onwards I think.

Back to Scripting SOAP

As I was kind of out of luck with using PowerCLI cmdlets,
I decided to write my own library,
that I would execute using PowerShell.

First command needs to shutdown my fileserver.
Issue:
hello, authenticate, shutdown

Used Burp suite to diagnose the http frames being sent received from/to vSphere client/ESXi.
I haven’t used this tool before, but it gave very good visibility of the messages being sent/received.
The vSphere client has a config file here:
C:\Program Files\VMware\Infrastructure\Virtual Infrastructure Client\Launcher\VpxClient.exe.config
that you can change the ports that the vSphere client sends/receives on,
but I found it easier to just set the IP address / Name field in the GUI
to point to 127.0.0.1:8080 This is where the Burp proxy listens on by default.

In Burp, you will also need to add another proxy listener to the proxy->options tab.
Set Local Listener to 8080,
Uncheck listen on loopback interface only,
Check support invisible proxying for non-proxy-aware clients.
The in-app help has good documentation on this.
Set redirect to host to the ESXi host name.
Set redirect to port to the ESXi’s default SSL port of 443
Select the generate CA-signed per-host certificates radio button.

I also made sure the new proxy rule was the only one running.
When Burp captures each frame, you can forward each one onto any one of the other tools in the suite.
This is a really nice tool.

My PowerOffUPSGuests.ps1 was about to significantly change too.
Running my PowerOffUPSGuests.ps1 script using PowerShell

PS C:\Scripts\UPS&gt; . ".\PowerOffUPSGuests.ps1"

We no longer need to pass any arguments to PowerOffUPSGuests.ps1

I was going to be using .net 4 libraries in my PowerOffUPSGuests.dll,
so needed to Let PowerShell know about the .net 4 CLR.
By default PS 2.0 is only aware of the .net 2.0 framework.

Some insight on this:
http://stackoverflow.com/questions/2094694/launch-PowerShell-under-net-4
http://tfl09.blogspot.com/2010/08/using-newer-versions-of-net-with.html
http://www.powergui.org/thread.jspa?threadID=13403

So needed to create a couple of config files for
%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\PowerShell.exe
and
PowerShell_ise.exe
with config appended
%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\PowerShell.exe.config
and
PowerShell_ise.exe.config
with the following contents:

<?xml version="1.0"?>
<configuration>
  <startup useLegacyV2RuntimeActivationPolicy="true">
     <supportedRuntime version="v4.0.30319"/>
     <supportedRuntime version="v2.0.50727"/>
   </startup>
</configuration>


This works for PowerShell, but not for PowerGUI (obviously) which I was using for debugging.


So If you still need PowerGUI you’ll have to add the registry hacks explained in the links above.
Remember to remove them once finished as they take affect system wide.

I also had some trouble with later versions of C# than 2.0 when compiling on the fly in PowerShell.
Although I was specifying the language.

Add-Type -Path $typePath -CompilerParameters $compilerParameters -Language csharpversion3


Found a workaround this bug here.

# add the block of code we call into
$code = [io.file]::ReadAllText((Join-Path -Path $scriptPath -ChildPath $powerOffUPSGuestsFile))
Add-Type $code -CompilerParameters $compilerParameters -Language CSharpVersion3


We’ll go over the library code in the third part of this series.

As it stands now, the C:\Scripts\UPS\PowerOff.bat looks like this

echo.
echo PowerOffUPSGuests.ps1 started at the following time: %time% &gt;&gt; C:\Scripts\UPS\Log.txt
"C:\WINDOWS\system32\WindowsPowerShell\v1.0\PowerShell.exe" C:\Scripts\UPS\PowerOffUPSGuests.ps1
echo PowerOffUPSGuests.ps1 finished at the following time: %time% &gt;&gt; C:\Scripts\UPS\Log.txt
echo.


The PowerOffUPSGuests.ps1 looks like this

Set-StrictMode -Version 2.0

# add the assembly that does the work.
Add-Type -Path C:\Scripts\UPS\PowerOffUPSGuests.dll

# instantiate a PowerOffUPSGuests instance
$powerOffUPSGuestsInstance = New-Object -TypeName BinaryMist.Networking.Infrastructure.PowerOffUPSGuests

Write-Host $powerOffUPSGuestsInstance.InitShutdownOfServers() -ForegroundColor Green

The essential files

Testing that everything works

I was unsure whether we were going to be able to get ESXi to cleanly shutdown it’s guest VM’s.
As I’d had some trouble with this previously.

I was thinking about writing a WCF service and client to shutdown windows guests for now.
The service being on the machine that needed to be cleanly shutdown.
Could use something like the following command line in the service.

shutdown.exe -m //MachineNeedingShutdown -t 10 -c "Shutting down due to UPS running on battery." -s -f

Wrapped in something like this…

Process shutdownMyBox = new Process();
shutdownMyBox.StartInfo.FileName = shutdown.exe;
shutdownMyBox.StartInfo.Arguments = "-m //MachineNeedingShutdown -t 10 -c \"Shutting down due to UPS running on battery.\" -s -f";
shutdownMyBox.Start();

I was sure there was a better way though.

The sequence of events I was thinking of was something like the following:

First we try to shutdown every VM guest, set vMGuestTimmer
If all VM guests shutdown
——try put host into maintenance mode, set timer.
——when in maintenance mode
———shutdown host
——if enter maintenance mode not successful within time set
———shutdown host
On vMGuestTimmer
——shutdown host

There was a better/easier way though

In the PCNS Web UI -> PowerChute->MachineName->Configure Events
You can set PowerOff.bat to run after 30 seconds, or for testing,
set it to something really small, so it runs the command file sooner.
Set the time that’s required for the command file to complete to 5 minutes.
Although I don’t think it matters that much, as long as there’s enough time to start the execution of the PowerShell script.
Once the script is running, we don’t care how long PCNS thinks it should wait, as it’s non blocking.

To test that pcns will run your batch file:
Just put some temporary script, something like the following in your
C:\Scripts\UPS\PowerOff.bat

time/T &gt;&gt; C:\Scripts\UPS\MyTest.txt

These are some links from APC to help get your PCNS command file running:
http://jpaa-en.apc.com/app/answers/detail/a_id/7712
http://jpaa-en.apc.com/app/answers/detail/a_id/2441
http://jpaa-en.apc.com/app/answers/detail/a_id/1175


What is needed for ESXi to shut down all machines cleanly?

Graceful shutdown work around for ESXi guests.

Also it’s important to make sure the root user of ESXi has the Administrator Role.

What is needed for Windows VM’s to shutdown cleanly?

First ascertain whether or not your VM is/isn’t being shutdown cleanly.
eventvwr is your friend.

The scripts that may play a part in the shutting down of the Windows VM’s.
If you have a look at the VMware Tools Properties->Scripts tab
You can see for the shutdown script, that it actually does nothing.
If you find that your Windows box is not shutting down cleanly…
Add a custom script to the “Shut Down Guest Operating System” Script Event
I just created a shutdown.bat with the following in it.

C:\Windows\System32\shutdown.exe -s -t 1

This cleared up any errors I was getting in my Windows7 logs.

What is needed for Linux VM’s to shutdown cleanly?

If you’re looking at Debian based systems…
View the relevant log that contains shutdown info.

sudo vi /var/log/messages

and

sudo vi /var/log/syslog

From command mode (that’s [Esc]) to show line numbers,

type

:set number

or

:set nu

To find the matches for “shutdown” (without quotes) ignoring case

sudo grep -i -n "shutdown" /var/log/messages

Or easier still…
Once the file’s open in vi,
From command mode

/shutdown

[n]            will repeat the search forward
[N]            will repeat the search in opposite direction

My Debian wheezy server wasn’t getting shutdown cleanly.
So tried to install vmware tools, but found the easier way was to use open-vm-tools
Added contrib to my /etc/apt/sources.list
Installed open-vm-tools open-vm-source
Had some trouble with the NZ repo for those packages, they were corrupt.
So renamed /etc/apt/apt.conf so apt-get wasn’t using my cached packages from apt-cacher.

sudo apt-get clean
sudo apt-get update
sudo apt-get install open-vm-tools open-vm-source

The scripts that may play a part in the shutting down of the Linux VM’s.
Read this link.
There are also vmware-tools scripts
Mine didn’t appear to do much, but my server was being shutdown cleanly now.


Shout out if anythings unclear.

In part three I’ll be going over the library I’ve written that actually does the work 😉

Preparing APC Smart-UPS 1500 for Critical Servers

June 16, 2011

Part one of a three part series

on Setting up a UPS solution, to enable clean shutdown of vital network components.

This post is essentially about setting up a Smart-UPS and it’s NMC (Network Management Card),
as the project I embarked upon was a little large for a single post.

Christchurch NZ used to have quite stable power,
but recent earthquakes we’ve been having have changed that.
Now we endure very unstable power.
This fact,
along with the fact that if my RAID arrays were being written to when a power outage occurred,
prompted me to get my A into G on this project.

For a while now I’ve been looking into setting up a UPS solution to support my critical servers.
I already had a couple of UPS’s
Liebert PowerSure 250 VA
Eaton Powerware 5110 500 VA
Both of which were a bit small to support a fairly hungry hypervisor, dedicated file server, 24 port Cisco catalyst switch and a home made router.
Also the FreeNAS (BSD ) driver for USB that was supposed to work with the 5110, didn’t seem to.
In considering the above; I had a couple of options.
With ESXi we can use an APC UPS and a network management card or the Powerware 5110 connected to a network USB hub
and a virtual guest listening to its events, ready to issue shutdown procedures as per James Pearce’s solution
but to any number of machines.

What I wanted was a single UPS plugged into a single box that would receive on battery events and do the work of shutting down the various machines listed (any type of machine, including virtual hosts and guests).
There didn’t appear to be a single piece of software that would do this, so I wrote it.
I’ll go over this in a latter post.

So I would need either a network connected USB hub. As explained here.

Simple Two Port Network Connected USB Hub

Hardware solutions and all work well from VM guests from what I’ve read.
AnywhereUSB from Digi …
USB server from Keyspan
USB Anywhere from Belkin

Software solutions, Need physical PC that has USB device/s plugged in.
USB@nywhere
USB over Network
USB Redirector

Or a network management card (something like the AP9606) for the UPS as explained here by James
Powerware 5110 doesn’t support a network management card, so only option I see for this UPS is a network connected USB hub.
APC SMART-UPS supports network management cards and I think these would be the best option for this UPS.

The AP9606

It was starting to look like an APC UPS would be the better option.
I had already been looking for one of these for quite a while, and I missed a couple of them.

The one I eventually picked up

APC Smart-UPS

Second hand APC Smart-UPS 1500
$200 + shipping = just under $300.
AP9606 NMC $50 + shipping = aprx $80.
So for $380 even if I needed a new battery ($250),
I still had a $1300 UPS, NMC not included, for $600.
Turned out the battery was fine,
so all up $380 to support a bunch of hardware.

You’ll need to give the card an IPv4 address that suites your subnet.
As my card was second hand, it already had one,
but I didn’t know what it was.
In order to give the card an IP, you have 2 obvious options

1.  serial cable and terminal emulator

2.  Ethernet and ARP

As I didn’t have the “special” serial cable,
I decided to go the Ethernet route.
I would need the MAC address.
The subnet mask and default gateway also need to be set up.
Pg 11 of APC_ap9606_installation_guide.pdf goes through the procedure.
All APC devices have a MAC address that begin with 00 C0 B7
Although my network management card had a sticker with the MAC address on it.
“You may want to check your DHCP client list for any MAC addresses beginning with 00 C0 B7,
which indicates an APC address.
In addition, check the card you are trying to configure.
Any card with valid IP settings will have a solid green status LED”.

When I received my AP9606 Web SNMP Management Card, I didn’t have a clue what the IP address had been set to.
If it was a new card it wouldn’t have yet been set and I would be able to easily set it without having to workout
what its subnet was.
On Pg 11 of the “Web/SNMP Management Card Installation Manual”
It goes through setting up an IP from scratch using ARP.
So I plugged my notebook into the AP9606’s Ethernet port and spun up Wireshark.

What you’ll generally be looking for is a record with the Source looking like

"American_[last 3 bytes of MAC]"

Time                    Source                Destination    Protocol    Info
231    715.948894    American_42:6f:b1    Broadcast    ARP        Who has 10.1.80.3?  Tell 10.1.80.222

And an ARP request that looks something like the following…
The first 3 bytes of the MAC will always be 00-C0-B7 for a AP9606.

Address Resolution Protocol (request)
 Sender MAC address: American_42:6f:b1 (00:c0:b7:[3 more octets here])
 Sender IP address: 10.1.80.222 (10.1.80.222)
 Target MAC address: American_42:6f:b1 (00:c0:b7:[3 more octets here])
 Target IP address: 10.1.80.3 (10.1.80.3)

I set the notebook to use a static IP of
10.1.80.2/24
and default gateway to the Target IP of
10.1.30.3
You may have to play around a bit with the subnet mask until you get it right.
I was just lucky.
Then tried using ARP to assign the new IP address,
but it wasn’t sticking.
So I tried to telnet in and was prompted for a username and password.
The default of apc for both was incorrect so obviously it had already been altered.
There is also another account of u- User p- apc
but this didn’t exist or had been changed.
So I contacted APC for the backdoor account as discussed here
and was directed to here.
This is no good unless you have a special serial cable which I didn’t.
I asked for the pin layout of the cable and was told,
that they make them,
but don’t know what the pin layout is.
The nice fellow at APC support directed me to a cable to buy.
A little pricey at $100NZ,
for a single use cable.
There is no proper way to reset the password by the Ethernet interface.
This left me with two obvious options.

1.  Make up a serial cable with I believe…

Pin#2 Female to Pin#2 Male,
Pin#3 Female to Pin#1 Male,
Pin#5 Female to Pin#9 Male,
and find a computer with a com port.
Layout info found here
It was correct.
That would cost next to nothing.

2.  just crack the credentials with one of these.

The second seemed like it would be the path of least resistance immediately (this turned out to be incorrect),
as I had the software, but not enough parts for a serial cable.
THC-Hydra seemed like a good option.
Once I downloaded and ran Hydra, I received the following error

 5 [main] ? (1988) C:\cygwin\bin\bash.exe: *** fatal error - system shared
 memory version mismatch detected - 0x75BE0074/0x75BE0096.
 This problem is probably due to using incompatible versions of the cygwin DLL.
 Search for cygwin1.dll using the Windows Start->Find/Search facility
 and delete all but the most recent version.  The most recent version *should*
 reside in x:\cygwin\bin, where 'x' is the drive on which you have
 installed the cygwin distribution.  Rebooting is also suggested if you
 are unable to find another cygwin DLL

This error is due to having incompatible versions of cygwin1.dll on your system.
So did a search for them and found that my SSH install had an older version of cygwin1.dll.
So renamed it,
and still had problems,
rebooted, and all was good.
the only cygwin1.dll should be in the same directory that hydra.exe is run from.

How to use THC-Hydra

Some good references here…
http://www.youtube.com/watch?v=kzJFPduiIsI
http://www.pauldotcom.com/2007/03/01/password_cracking_with_thchydr.html

Command I used.

C:\hydra-5.4-win>hydra -L logins.txt -P passwords.txt -e n -e s -o hydraoutput.txt -v 10.1.80.222 telnet "Welcome hacker"

I got a false positive of User name n/a Password steven
So rather than spend more time on populating the logins.txt and passwords.txt.

I decided to try the serial cable route

As it turned out, I wouldn’t have guessed the username,
found this out once I logged on using the serial interface.
This is the pinout I used.

This is the single use cable I made.
Total cost of $0.00


Make sure you’re all plugged in.
I used minicom as my terminal emulator to connect to the UPS’s com port.
Installation and usage details here.

You need to make sure you’re serial port/s are on in the BIOS.
I didn’t check mine, but they were on.

Need to make sure Linux knows about your serial port/s
Run the following command:

Use setserial to provide the configuration information associated with your serial ports.

Configuring your serial ports.

To setup your terminal emulator (minicom in my case):

$ minicom -s -c on

Choose “Serial port setup”
and you will be presented with a menu like the following.

This is where you get to set the following:
2400 BPS, 8 databits, No parity,
one stop bit and flow control is set to none.
Then select Save setup as dfl

Exit.

You should now be prompted for authentication from the Smart-Ups.

Or you can choose “Exit from Minicom” and run

$ minicom -c on

later.
If you get output like…

Device /dev/ttyS[number of your port here] is locked.

You’ll have to

# rm /var/tmp/LOCK..ttyS[number of your port here]

Now is where you get to log on as the default user/pass apc/apc
Press the reset button on the AP9606
and press Enter key,
then repeatedly if necessary.
This is poking the AP9606 in order to get a login prompt
Once you get the User Name,
you can enter the “apc” user (without the quotes) and then for the Password,
“apc” (without the quotes).
You have a 30 second window here to login.
Else you have to repeat the reset process and try again.

From the Control Console menu,
select System, then User Manager.
Select Administrator,
and change the User Name and Password settings,
both of which are currently apc.

I also changed the IP settings.
From the Control Console,
select
2- Network
1- TCP/IP
and change your IP settings.

There are quite a few settings you can change on the card,
you should just be able to follow your nose from here.
You’ll also want to make sure the Web Access is Enabled.
Take note of the port also, usually 8000.
Changing the password via the serial interface is also detailed here.
This post was also quite helpful.

Changed the IP settings back to how they were on my notebook.
Could now connect via telnet and HTTP.
Turned md5 on to try and boost the security of passing credentials to the web UI.
Turned out the jre is also needed for this.
Went through that process and it was looking promising,
but the web UI no longer accepted my password.
Not sure why this is,
but it means if you want to be secure when you log into the web UI,
you are going to have to plug your Ethernet cable directly into the AP9606.
Otherwise your passing credentials in plan text.

Upgrade of firmware

The latest firmware is found here.
Directions on upgrading are found here.
In saying that, APC recommended I use the earlier aos325.bin and sumx326.bin from here if using Windows XP.
Some details around the firmware required for the different management card types for use in a Smart Slot equipped APC UPS
The firmware version is found under Help->About System on the NMC’s Web interface.