Followed up with a test deployment and drive.
The best time to install a HIDS is on a fresh install before you open the host up to the internet or even your LAN if it’s corporate. Of course if you don’t have that luxury, there are a bunch of tools that can help you determine if you’re already owned. Be sure to run one or more over your target system before your HIDS bench-marks it.
The reason I chose Stealth and OSSEC to take further into an evaluation was because they rose to the top of a preliminary evaluation I performed during a recent Debian Web server hardening process where I looked at several other HIDS as well.
ossec-hids on github
- Many developers, contributors, managers, reviewers, translators (infact the OSSEC team looks almost as large as the Stealth user base)
Lots of documentation. Not always the easiest to navigate. Lots of buzz on the inter-webs.
- Lots of documentation (so far looking good)
- The manual
- Mailing list
- Several good looking books
- Commercial Support
Community / Communication
IRC channel #ossec at irc.freenode.org Although it’s not very active.
- Manager (sometimes called server): does most of the work monitoring the Agents. It stores the file integrity checking databases, the logs, events and system auditing entries, rules, decoders, major configuration options.
- Agents: small collections of programs installed on the machines we are interested in monitoring. Agents collect information and forward it to the manager for analysis and correlation.
There are quite a few other ancillary components also.
You can also go the agent-less route which may allow the Manager to perform file integrity checks using agent-less scripts. As with Stealth, you’ve still got the issue of needing to be root in order to read some of the files.
Agents can be installed on VMware ESX but from what I’ve read it’s quite a bit of work.
Features (What does it do)
- File integrity checking
- Rootkit detection
- Real-time log file monitoring and analysis (you may already have something else doing this)
- Intrusion Prevention System (IPS) features as well: blocking attacks in real-time
- Alerts can go to a databases MySQL or PostgreSQL or other types of outputs
- There is a PHP web UI that runs on Apache if you would rather look at pretty outputs vs log files.
What I like
To me, the ability to scan in real-time off-sets the fact that the agents need binaries installed. This hinders the attacker from covering their tracks.
Backed by a large company Trend Micro.
Options. Install options for starters. You’ve got the options of:
- Agent-less installation as described above
- Local installation: Used to secure and protect a single host
- Agent installation: Used to secure and protect hosts while reporting back to a
central OSSEC server
- Server installation: Used to aggregate information
Can install a web UI on the manager, so you need Apache, PHP, MySQL.
What I don’t like
- Unlike Stealth, The fact that something has to be installed on the agents
- The packages are not in the standard repositories. The downloads, PGP keys and directions are here.
- I think Ossec may be doing to much and if you don’t like the way it does one thing, you may be stuck with it. Personally I really like the idea of a tool doing one thing, doing it well and providing plenty of configuration options to change the way it does it’s one thing. This provides huge flexibility and minimises your dependency on a suite of tools and/or libraries
- Information overload. There seems to be a lot to get your head around to get it set-up. There are a lot of install options documented (books, interwebs, official docs). It takes a bit to workout exactly the best procedure for your environment. Following are some of the resources I used:
Who’s Behind Stealth
Author: Frank B. Brokken. An admirable job for one person. Frank is not a fly-by-nighter though. Stealth was first presented to Congress in 2003. It’s still actively maintained and used by a few. It’s one of GNU/Linux’s dirty little secrets I think. It’s a great idea, makes a tricky job simple and does it in an elegant way.
- 3.00.00 (2014-08-29)
- 2.11.03 (2013-06-18)
- Once you install Stealth, all the documentation can be found by
sudo updatedb && locate stealth. I most commonly used: HTML docs (/usr/share/doc/stealth-doc/manual/html/) and (/usr/share/doc/stealth-doc/manual/pdf/stealth.pdf) for easy searching across the HTML docs
- man page (/usr/share/doc/stealth/stealthman.html)
- Examples: (/usr/share/doc/stealth/examples/)
- Once you install Stealth, all the documentation can be found by
- More covered in my demo set-up below
- 3.00.00 (2014-08-29) is in the repository for Debain Jessie
- 2.11.03-2 (2013-06-18) This is as recent as you can go for Linux Mint Qiana (17) and Rebecca (17.1) within the repositories, unless you want to go out of band. There are quite a few dependencies
lddwill show you:
libbobcat.so.3 =&gt; /usr/lib/libbobcat.so.3 (0xb765d000) libpthread.so.0 =&gt; /lib/i386-linux-gnu/i686/cmov/libpthread.so.0 (0xb7641000) libstdc++.so.6 =&gt; /usr/lib/i386-linux-gnu/libstdc++.so.6 (0xb754e000) libm.so.6 =&gt; /lib/i386-linux-gnu/i686/cmov/libm.so.6 (0xb7508000) libgcc_s.so.1 =&gt; /lib/i386-linux-gnu/libgcc_s.so.1 (0xb74eb000) libc.so.6 =&gt; /lib/i386-linux-gnu/i686/cmov/libc.so.6 (0xb7340000) libX11.so.6 =&gt; /usr/lib/i386-linux-gnu/libX11.so.6 (0xb71ee000) libcrypto.so.1.0.0 =&gt; /usr/lib/i386-linux-gnu/i686/cmov/libcrypto.so.1.0.0 (0xb7022000) libreadline.so.6 =&gt; /lib/i386-linux-gnu/libreadline.so.6 (0xb6fdc000) libmilter.so.1.0.1 =&gt; /usr/lib/i386-linux-gnu/libmilter.so.1.0.1 (0xb6fcb000) /lib/ld-linux.so.2 (0xb7748000) libxcb.so.1 =&gt; /usr/lib/i386-linux-gnu/libxcb.so.1 (0xb6fa5000) libdl.so.2 =&gt; /lib/i386-linux-gnu/i686/cmov/libdl.so.2 (0xb6fa0000) libtinfo.so.5 =&gt; /lib/i386-linux-gnu/libtinfo.so.5 (0xb6f7c000) libXau.so.6 =&gt; /usr/lib/i386-linux-gnu/libXau.so.6 (0xb6f78000) libXdmcp.so.6 =&gt; /usr/lib/i386-linux-gnu/libXdmcp.so.6 (0xb6f72000)
- 2.10.01 (2012-10-04) is in the repository for Debian Wheezy
Community / Communication
No community really. I see it as one of the dirty little secretes that I’m surprised many diligent sysadmins haven’t jumped on. The author is happy to answer emails. The author doesn’t market.
The computer initiating the scan.
Needs two kinds of outgoing services:
- ssh to reach the clients
- mail transport agent (MTA)(sendmail, postfix)
Considerations for the controller:
- No public access.
- All inbound services should be denied.
- Access only via its console.
- Physically secure location (one would think goes without saying, but you may be surprised).
- Sensitive information of the clients are stored on the controller.
- Password-less access to the clients for anyone who gains controller root access, unless ssh-cron is used, which appears to have been around since 2014-05.
The computer/s being scanned. I don’t see any reason why a Stealth solution couldn’t be set-up to look after many clients.
The controller stores one to many policy files. Each of which is specific to a single client and contains use directives and commands. It’s recommended policy to take copies of the client programmes such as the hashing programme sha1sum, find and others that are used extensively during the integrity scans and copy them to the controller to take bench-mark hashes. Subsequent runs will do the same to compare with the initial hashes stored.
Features (What does it do)
File integrity tests leaving virtually no sediments on the tested client.
Stealth subscribes to the “dark cockpit” approach. I.E. no mail is sent when no changes were detected. If you have a MTA, Stealth can be configured to send emails on changes it finds.
What I like
- It’s simplicity. There is one package to install on the controller. Nothing to install on the client machines. Client just needs to have the controllers SSH public key. You will need a Mail Transfer Agent on your controller if you don’t already have one. My test machine (Linux Mint) didn’t have one.
- Rather than just modifying the likes of sha1sum on the clients that Stealth uses to perform it’s integrity checks, Stealth would somehow have to be fooled into thinking that the changed hash of the sha1sum it’s just copied to the controller is the same as the previously recorded hash that it did the same with. If the previously recorded hash is removed or does not match the current hash, then Stealth will fire an alert off.
- It’s in the Debian repositories. Which is a little surprising considering I couldn’t find any test suite results.
- The whole idea behind it. Systems being monitored give little appearance that they’re being monitored, other than I think the presence of a single SSH login when Stealth first starts in the auth.log. This could actually be months ago, as the connection remains active for the life of Stealth. The login could be from a user doing anything on the client. It’s very discrete.
- unpredictability of Stealth runs is offered through Stealth’s
--repeat 60 --random-interval 30results in new Stealth-runs on average every 75 seconds.
- Subscribes to the Unix philosophy: “do one thing and do it well”
- Stealth’s author is very approachable and open. After talking with Frank and suggesting some ideas to promote Stealth and it’s community, Frank started a discussion list.
What I don’t like
- Lack of visible code reviews and testing.Yes it’s in Debian, but so was OpenSSL and Bash
- One man band. Support provided via one person alone via email. Comparing with the likes of Ossec which has …
- Lack of use cases. I don’t see anyone using/abusing it. Although Frank did send me some contacts of other people that are using it, so again, a very helpful author. Can’t find much on the interwebs. The documentation has clear signs that it’s been written and is targeting people already familiar with the tool. This is understandable as the author has been working on this project for nine years and could possibly be disconnected with what’s involved for someone completely new to the project to dive in and start using it. In saying that, that’s what I did and so far it worked out well.
- This tells me that either very few are using it or it has no bugs and the install and configuration is stupidly straight forward or both.
- Small user base. This is how many people are happy to reveal they are using Stealth.
- Reading through the userguide, the following put me off a little: “preferably the public ssh-identity key of the controller should be placed in the client’s root .ssh/authorized_keys file, granting the controller root access to the client. Root access is normally needed to gain access to all directories and files of the client’s file system.” I never allow SSH root access to servers. So I’m not about to start. What’s worse, is that Stealth SSHing from server to client with key-pair can only do so automatically if the pass-phrase is blank. If it’s not blank then someone has to drive Stealth and the whole idea of Stealth (as far as I can tell) is to be an automatic file integrity checker.
There are however some work-arounds to this. ssh-cron can run scheduled Stealth jobs, but needs to aquire the pass-phrase from the user once. It does this via ssh-askpass which is a X11 based pass-phrase input dialog. If your running ssh-cron and Stealth from a server (which would be a large number of potential users I would think) you won’t have X11. So if that’s the case ssh-cron is out of the question. At least that’s how I understand it from the man page and emails from Frank. Frank mentioned in an email: “Stealth itself could perfectly well be started `by hand’ setting up the connection using, e.g., an existing ssh private-key, which could thereafter completely be removed from the system running Stealth. The running Stealth process would then continue to use the established connection for as long as it’s running. This may be a very long time if the –daemon option is used.” I’ve used Monit to do any checks on the client that need root access. This way Stealth can run as a low privileged user.
- In order to use an SSH key-pair with passphrase and have your controller resume scans on reboot, you’re going to need ssh-cron. Some distros like Mint and Ubuntu only have very old versions of libbobcat (3.19.01 (December 2013)) in their repositories. You could re-build if you fancy the dependency issues it may bring. Failing that, use Debian which is way more up to date, or just fire stealth off each time you reboot your controller machine manually and run it as a daemon with arguments such as
--daemonif running stealth >= 4.00.00),
--repeat.This will cause Stealth to keep running (sleeping) most of the time, then doing it’s checks at the intervals you define.
In making all of my considerations, I changed my mind quite a few times on which offerings were most suited to which environments. I think this is actually a good thing, as I think it means my evaluations were based on the real merits of each offering rather than any biases.
If you don’t already have real-time logging to an off-site syslog server, then OSSEC can help here.
The simplicity of Stealth, flatter learning curve and it’s over-all philosophy is what won me over.
Stealth Up and Running
I tested this out on a Mint installation.
Installed stealth and stealth-doc (2.11.03-2) via synaptic package manager. Then just did a
locate for stealth to find the docs and other example files. The following are the files I used for documentation, how I used them and the tab order that made sense to me:
- The main documentation index: file:///usr/share/doc/stealth-doc/manual/html/stealth.html
- Chapter one introduction: file:///usr/share/doc/stealth-doc/manual/html/stealth01.html
- Chapter three to help build up a policy file: file:///usr/share/doc/stealth-doc/manual/html/stealth03.html
- Chapter five for running Stealth and building up policy file: file:///usr/share/doc/stealth-doc/manual/html/stealth05.html
- Chapter six for running Stealth: file:///usr/share/doc/stealth-doc/manual/html/stealth06.html
- Chapter seven for arguements to pass to Stealth: file:///usr/share/doc/stealth-doc/manual/html/stealth07.html
- Chapter eight for error messages: file:///usr/share/doc/stealth-doc/manual/html/stealth08.html
- The Man page: file:///usr/share/doc/stealth/stealthman.html
- Policy file examples: file:///usr/share/doc/stealth/examples/
- Useful scripts to use with Stealth: file:///usr/share/doc/stealth/scripts/usr/bin/
- All of the documentation in simple text format (good for searching across chapters for strings): file:///usr/share/doc/stealth-doc/manual/text/stealth.txt
Files I would need to copy and modify were:
Files I used for reference to build up a policy file:
As mentioned above, providing you have a working MTA, then Stealth will just do it’s thing when you run it. The next step is to schedule it’s runs. This can be also (as mentioned above) with a pseudo random interval.
Feel free to leave a comment if you need help setting Stealth up, as it did take a bit of fiddling, but does what it says it does on the box very well.