Archive for the ‘Uncategorized’ Category

The Cloud Shared Responsibility Model

October 2, 2017

Risks

The shared responsibility model is one that many have not grasped or understood well. Let’s look at the responsibilities of the parties.

CSP Responsibility

The CSP takes care of the infrastructure, not the customer specific configuration of it, and Due to the shear scale of what they are building, are able to build in good security controls, in contrast to the average system administrator, which just does not have the resources or ability to focus on security to the same degree.

Due to the share scale, the average CSP has a concentrated group of good security professionals vs a business who’s core business is often not closely related to security. So CSPs do provide good security mechanisms, but the customer has to know and care enough to use them.

CSPs creating the infrastructural architecture, building the components, frameworks, hardware, platform software in most cases are taking security seriously and doing a reasonable job.

CSP Customer Responsibility

CSP customers are expected to take care of their own security in terms of:

  1. Their people working with the technology
  2. Application security, ultimately leading back to shortcomings in people: Lack of skills, experience, engagement, etc.
  3. Configuring the infrastructure and/or platform components: Again leading back to people defects

but all to often the customers responsibility is neglected, which renders The Cloud no better for the customer in terms of security.

The primary problem with The Cloud is: Customers have the misconception that someone else is taking care of all their security. That is not how the shared responsibility model works though. Yes the CSP is probably taking care of the infrastructure security, but other forms of security such as I just listed above, are even more important than before the shift to The Cloud, this is because these items are now the lowest hanging fruit for the attacker.

The following are a set of questions (verbatim) I have been asked recently, and that I hear similar versions of frequently:

  • As a software engineer, do I really care about physical network security and network logging?
  • Surely “as a software engineer”, I can just use TLS and that is the end of it?
  • Well if the machine is compromised, then we give up on security, we aren’t responsible for the network
  • What is the difference between application security and network security? Aren’t they just two aspects of the same thing?
  • If I have implemented TLS for communication, have I fixed all of the network security problems?

Countermeasures

The following responsibilities are those that you need to have a good understanding of in order to establish a good level of security when operating in The Cloud.

CSP Responsibility

There is not a lot you can do about this, just be aware of what you are buying into before you do so. AWS for example states: “Customers retain control of what security they choose to implement to protect their own content, platform, applications, systems and networks, no differently than they would for applications in an on-site datacenter.

CSP Customer Responsibility

If you leverage The Cloud, Make sure the following aspects of security are all at an excellent level:

  1. People security: Discussed in Fascicle 0 under the People chapter
  2. Application security: Discussed in the Web Applications chapter. The move to application security was also discussed in the VPS chapter as a response of using Docker containers
  3. Configuring the infrastructure and/or platform components: Usually CSP specific, but I cover some aspects in this chapter

The following is in response to the set of frequently asked questions under the risks subsection of CSP Customer Responsibility:

  • (Q): As a software engineer, do I really care about physical network security and network logging?
    (A): In the past, many aspects of network security were the responsibility of the Network Administrators, with the move to The Cloud, this has to large degree changed. The networks established (intentionally or not) between the components we are leveraging and creating in The Cloud are a result of Infrastructure and Configuration Management, often (and rightly so) exp3ressed as code. Infrastructure as Code (IaC). As discussed in the Network Security subsection, this is now the responsibility of the Software Engineer
  • (Q): Surely “as a software engineer”, I can just use TLS and that is the end of it?
    (A): TLS is one very small area of network security. Its implementation as HTTPS and the PKI model is effectively broken. If TLS is your only saviour, putting it bluntly, you are without hope. The Network Chapter covers the tip of the network security ice berg, network security is a huge topic, and one that has many books written along with other resources that provide more in-depth coverage than I can provide as part of a holistic view of security for Software Engineers. Software Engineers must come to grips with the fact that they need to implement defence in depth
  • (Q): Well if the machine is compromised, then we give up on security, we aren’t responsible for the network
    (A): For this statement, please refer to the VPS chapter for your responsibilities as a Software Engineer in regards to “the machine”. In regards to “the network”, please refer to the Network Security subsection
  • (Q): What is the difference between application security and network security? Aren’t they just two aspects of the same thing?
    (A): No, for application security, see the Web Applications chapter. For network security, see the Network chapter. Again, as Software Engineers, you are now responsible for all aspects of information security
  • (Q): If I have implemented TLS for communication, have I fixed all of the network security problems?
    (A): If you are still reading this, I’m pretty sure you know the answer, please share it with other Developers, Engineers as you receive the same questions
Advertisements

Holistic Info-Sec for Web Developers F1: Content Complete

September 12, 2017

2017-09-11

Fascicle 1 is now content complete
Weighing in at aprox 550 pages incl Additional Resources and Attributions

  • Added links to Network Security Interview between Kim Carter and Haroon Meer on Software Engineering Radio … to be released in a day or two
  • Updated threat tags
  • Code formatting changes
  • Punctuation modifications

Cloud

Ready for technical review
Strong focus on AWS, although other CSPs discussed
50 Pages of content added

  • Shared Responsibility Model: CSP Responsibility, CSP Customer Responsibility
  • CSP Evaluation
  • Cloud Service Provider vs In-house
    • Skills
    • EULA
    • Giving up Secrets
    • Location of Data
    • Vendor lock-in
    • Possible Single Points of Failure
  • People Sec
  • App Sec
  • Net Sec
  • Violations of Least Privilege
  • Storage of Secrets
    • Private Key Abuse: SSH, TLS
    • Credentials and Other Secrets
      • Entered by People
      • Entered by Software: HashiCorp Vault, Docker secrets, Ansible Vault, AWS Key Management Service and Parameter Store
  • Serverless
    • Third Party Services
    • Perimeterless
    • Functions
    • DoS of Lambda Functions
  • Infrastructure and Configuration Management

Web Applications

  • Updated OWASP Top 10 resources to 2017
  • Added AWS WAF

Additional Resources

  • Getting Secrets out of Docker images
  • Password Managers For Business Use
  • Many tooling options covered

Attributions

  • Thinkst tools (Canary tools and tokens)
  • DropboxC2C for Data Exfiltration, Infiltration
  • Hosting providers forced to give up customer secrets
  • Software Engineering Radio show on Network Security with host: Kim Carter, guest: Haroon Meer
  • Docker Image layers
  • AWS Lambda

Many other attributions added

Holistic Info-Sec for Web Developers (F1)(VPS, Network, Cloud, Web Applications)

Github

Holistic Info-Sec for Web Developers F1 Large update to VPS chapter

October 7, 2016

Holistic Info-Sec for Web Developers (F1)(VPS, Network, Cloud, Web Applications)

Git Changeset

Large number of image updates due to finding that many were not up to scratch when Fascicle 0 went to print.
Swapped text images for real images.

Many large additions to the VPS chapter and fewer to the Network chapter, such as:
* The pitfalls of logging within networks and some ideas and implementations on how to overcome
* Disabling, removing and hardening the services of a VPS
* Granular OS partitioning and locking down the mounting of partitions
* Caching apt packages for all VPS
* Reviewing VPS password strategies and making the most suitable modifications to achieve enough security for you
* Disabling root logins on as many of the consoles as possible
* SSH, Symmetric and Asymmetric crypto-systems and their place in SSH
* The ciphers used in SSH, pros, cons, some history
* Hashing and its application in SSH
* How the SSH connection procedure works
* Hardening SSH
* Configuring which hosts may access your server
* SSH Key-pair authentication
* Techniques for tunneling SSH
* Understanding enough about NFS to produce exports that will suite your environmental security concerns
* Some quick commands to provide visibility as to who is doing what and when on your servers
* VPS logging and alerting: We look at a large number of options available and the merits of them
* Managing your logs effectively, so that they will be around when you need them and not tampered with. We work through transferring them off-site in real-time. We address reliability, resilience, integrity, connectivity of the proposed solutions. Verifying that the logs being transferred are in-fact encrypted.
* Proactive server monitoring, discuss goals, and the evaluation criteria for the offerings that were evaluated
* Implementation of proactive server monitoring, what works well, what doesn’t
* Keeping your (NodeJS) applications not just running, but healthy
* We discuss the best of bread HIDS/HIPS, then go on to implement the chosen solution
* Made a start with Docker insecurities and mitigation’s.
* Quick discussion around host firewalls
* Preparing DMZ and your VPS for the DMZ
* Additional Web Server preparation
* Deployment options
* Post DMZ deployment considerations

Captcha Considerations

December 31, 2015

Risks

Exploiting Captcha

Lack of captchas are a risk, but so are captchas themselves…

Let’s look at the problem here? What are we trying to stop with captchas?

Bots submitting. What ever it is, whether:

  • Advertising
  • Creating an unfair advantage over real humans
  • Link creation in attempt to increase SEO
  • Malicious code insertion

You are more than likely not interested in accepting it.

What do we not want to block?

People submitting genuinely innocent input. If a person is prepared to fill out a form manually, even if it is spam, then a person can view the submission and very quickly delete the validated, filtered and possibly sanitised message.

Countermeasures

PreventionVERYEASY

Types

Text Recognition

recaptcha uses this technique. See below for details.

Image Recognition

Uses images which users have to perform certain operations on, like dragging them to another image. For example: “Please drag all cat images to the cat mat.”, or “Please select all images of things that dogs eat.” sweetcaptcha is an example of this type of captcha. This type completely rules out the visually impaired users.

Friend Recognition

Pioneered by… you guessed it. Facebook. This type of captcha focusses on human hackers, the idea being that they will not know who your friends are.

Instead of showing you a traditional captcha on Facebook, one of the ways we may help verify your identity is through social authentication. We will show you a few pictures of your friends and ask you to name the person in those photos. Hackers halfway across the world might know your password, but they don’t know who your friends are.

I disagree with that statement. A determined hacker will usually be able to find out who your friends are. There is another problem, do you know who all of your friends are? Every acquaintance? I am terrible with names and so are many people. This is supposed to be used to authenticate you. So you have to be able to answer the questions before you can log in.

Logic Questions

This is what textcaptcha uses. Simple logic questions designed for the intelligence of a seven year old child. These are more accessible than image and textual image recognition, but they can take longer than image recognition to answer, unless the user is visually impared. The questions are usually language specific also, usually targeting the English language.

User Interaction

This is a little like image recognition. Users have to perform actions that virtual intelligence can not work out… yet. Like dragging a slider a certain number of notches.
If an offering gets popular, creating some code to perform the action may not be that hard and would definitely be worth the effort for bot creators.
This is obviously not going to work for the visually impaired or for people with handicapped motor skills.

 

In NPM land, as usual there are many options to choose from. The following were the offerings I evaluated. None of which really felt like a good fit:

Offerings

  • total-captcha. Depends on node-canvas. Have to install cairo first, but why? No explanation. Very little of anything here. Move on. How does this work? Do not know. What type is it? Presume text recognition.
  • easy-captcha is a text recognition offering generating images
  • simple-captcha looks like another text recognition offering. I really do not want to be writing image files to my server.
  • node-captcha Depends on canvas. By the look of the package this is another text recognition in a generated image.
  • re-captcha was one of the first captcha offerings, created at the Carnegie Mellon University by Luis von Ahn, Ben Maurer, Colin McMillen, David Abraham and Manuel Blum who invented the term captcha. Google later acquired it in September 2009. recaptcha is a text recognition captcha that uses scanned text that optical character recognition (OCR) technology has failed to interpret, which has the added benefit of helping to digitise text for The New York Times and Google Books.
    recaptcha
  • sweetcaptcha uses the sweetcaptcha cloud service of which you must abide by their terms and conditions, requires another node package, and requires some integration work. sweetcaptcha is an image recognition type of captcha.
    sweetcaptcha
  • textcaptcha is a logic question captcha relying on an external service for the questions and md5 hashes of the correct lower cased answers. This looks pretty simple to set up, but again expects your users to use their brain on things they should not have to.

 

After some additional research I worked out why the above types and offerings didn’t feel like a good fit. It pretty much came down to user experience.

Why should genuine users/customers of your web application be disadvantaged by having to jump through hoops because you have decided you want to stop bots spamming you? Would it not make more sense to make life harder for the bots rather than for your genuine users?

Some other considerations I had. Ideally I wanted a simple solution requiring few or ideally no external dependencies, no JavaScript required, no reliance on the browser or anything out of my control, no images and it definitely should not cost any money.

Alternative Approaches

  • Services like Disqus can be good for commenting. Obviously the comments are all stored somewhere in the cloud out of your control and this is an external dependency. For simple text input, this is probably not what you want. Similar services such as all the social media authentication services can take things a bit too far I think. They remove freedoms from your users. Why should your users be disadvantaged by leaving a comment or posting a message on your web application? Disqus tracks users activities from hosting website to website whether you have an account, are logged in or not. Any information they collect such as IP address, web browser details, installed add-ons, referring pages and exit links may be disclosed to any third party. When this data is aggregated it is useful for de-anonymising users. If users choose to block the Disqus script, the comments are not visible. Disqus has also published its registered users entire commenting histories, along with a list of connected blogs and services on publicly viewable user profile pages. Disqus also engage in add targeting and blackhat SEO techniques from the websites in which their script is installed.
  • Services like Akismet and Mollom which take user input and analyse for spam signatures. Mollom sometimes presents a captcha if it is unsure. These two services learn from their mistakes if they mark something as spam and you unmark it, but of course you are going to have to be watching for that. Matt Mullenweg created Akismet so that his mother could blog in safety. “His first attempt was a JavaScript plugin which modified the comment form and hid fields, but within hours of launching it, spammers downloaded it, figured out how it worked, and bypassed it. This is a common pitfall for anti-spam plugins: once they get traction“. My advice to this is not to use a common plugin, but to create something custom. I discuss this soon.

The above solutions are excellent targets for creating exploits that will have a large pay off due to the fact that so many websites are using them. There are exploits discovered for these services regularly.

Still not cutting it

Given the fact that many clients count on conversions to make money, not receiving 3.2% of those conversions could put a dent in sales. Personally, I would rather sort through a few SPAM conversions instead of losing out on possible income.

Casey Henry: Captchas’ Effect on Conversion Rates

Spam is not the user’s problem; it is the problem of the business that is providing the website. It is arrogant and lazy to try and push the problem onto a website’s visitors.

Tim Kadlec: Death to Captchas

User Time Expenditure

Recording how long it takes from fetch to submit. This is another technique, in which the time is measured from fetch to submit. For example if the time span is under five seconds it is more than likely a bot, so handle the message accordingly.

Bot Pot

Spamming bots operating on custom mechanisms will in most cases just try, then move on. If you decide to use one of the common offerings from above, exploits will be more common, depending on how wide spread the offering is. This is one of the cases where going custom is a better option. Worse case is you get some spam and you can modify your technique, but you get to keep things simple, tailored to your web application, your users needs, no external dependencies and no monthly fees. This is also the simplest technique and requires very little work to implement.

Spam bots:

  • Love to populate form fields
  • Usually ignore CSS. For example, if you have some CSS that hides a form field and especially if the CSS is not inline on the same page, they will usually fail at realising that the field is not supposed to be visible.

So what we do is create a field that is not visible to humans and is supposed to be kept empty. On the server once the form is submitted, we check that it is still empty. If it is not, then we assume a bot has been at it.

This is so simple, does not get in the way of your users, yet very effective at filtering bot spam.

Client side:

form .bot-pot {
   display: none;
}
<form>
   <!--...-->
   <div>
      <input type="text" name="bot-pot" class="bot-pot">
   </div>
   <!--...-->
</form>

Server side:

I show the validation code middle ware of the route on line 30 below. The validation is performed on line 16

var form = require('express-form');
var fieldToValidate = form.field;
//...

function home(req, res) {
   res.redirect('/');
}

function index(req, res) {
   res.render('home', { title: 'Home', id: 'home', brand: 'your brand' });
}

function validate() {
   return form(
      // Bots love to populate everything.
      fieldToValidate('bot-pot').maxLength(0)
   );
}

function contact(req, res) {

   if(req.form.isValid)
      // We know the bot-pot is of zero length. So no bots.
   //...
}

module.exports = function (app) {
   app.get('/', index);
   app.get('/home', home);
   app.post('/contact', validate(), contact);
};

So as you can see, a very simple solution. You could even consider combining the above two techniques.

Keeping Your Linux Server/s In Time With Your Router

March 28, 2015

Your NTP Server

With this set-up, we’ve got one-to-many Linux servers in a network that all want to be synced with the same up-stream Network Time Protocol (NTP) server/s that your router (or what ever server you choose to be your NTP authority) uses.

On your router or what ever your NTP server host is, add the NTP server pools. Now how you do this really depends on what your using for your NTP server, so I’ll leave this part out of scope. There are many NTP pools you can choose from. Pick one or a collection that’s as close to you’re NTP server as possible.

If your NTP daemon is running on your router, you’ll need to decide and select which router interfaces you want the NTP daemon supplying time to. You almost certainly won’t want it on the WAN interface (unless you’re a pool member) if you have one on your router.

Make sure you restart your NTP daemon.

Your Client Machines

If you have ntpdate installed, /etc/default/ntpdate says to look at /etc/ntp.conf which doesn’t exist without ntp being installed. It looks like this:

# Set to "yes" to take the server list from /etc/ntp.conf, from package ntp,
# so you only have to keep it in one place.
NTPDATE_USE_NTP_CONF=yes

but you’ll see that it also has a default NTPSERVERS variable set which is overridden if you add your time server to /etc/ntp.conf. If you enter the following and ntpdate is installed:

dpkg-query -W -f='${Status} ${Version}\n' ntpdate

You’ll get output like:

install ok installed 1:4.2.6.p5+dfsg-3ubuntu2

Otherwise install it:

apt-get install ntp

The public NTP server/s can be added straight to the bottom of the /etc/ntp.conf file, but because we want to use our own NTP server, we add the IP address of our server that’s configured with our NTP pools to the bottom of the file.

server <IP address of your local NTP server here>

Now if your NTP daemon is running on your router, hopefully you have everything blocked on its interface/s by default and are using a white-list for egress filtering.

In which case you’ll need to add a firewall rule to each interface of the router that you want NTP served up on.

NTP talks over UDP and listens on port 123 by default.

After any configuration changes to your ntpd make sure you restart it. On most routers this is done via the web UI.

On the client (Linux) machines:

sudo service ntp restart

Now issuing the date command on your Linux machine will provide the current time, yes with seconds.

Trouble-shooting

The main two commands I use are:

sudo ntpq -c lpeer

Which should produce output like:

            remote                       refid         st t when poll reach delay offset jitter
===============================================================================================
*<server name>.<domain name> <upstream ntp ip address> 2  u  54   64   77   0.189 16.714 11.589

and the standard NTP query program followed by the as argument:

ntpq

Which will drop you at ntpq’s prompt:

ntpq> as

Which should produce output like:

ind assid status  conf reach auth condition  last_event cnt
===========================================================
  1 15720  963a   yes   yes  none  sys.peer    sys_peer  3

Now in the first output, the * in front of the remote means the server is getting it’s time successfully from the upstream NTP server/s which needs to be the case in our scenario. Often you may also get a refid of .INIT. which is one of the “Kiss-o’-Death Codes” which means “The association has not yet synchronized for the first time”. See the NTP parameters. I’ve found that sometimes you just need to be patient here.

In the second output, if you get a condition of reject, it’s usually because your local ntp can’t access the NTP server you set-up. Check your firewall rules etc.

Now check all the times are in sync with the date command.

Culture in the work place

April 26, 2014

What is Climate?

The ups and downs, the hot and cold.
It’s easier to change than the culture.
The mood of an organisation can be seasonal which fluctuates more than a culture.
Refers to perceptions of organizational practices reported by people who work there (Rousseau 1988). Describes the work setting by those directly involved with it.

  • Communication: How open are people?
  • Dealing with conflict: Is it constructive or dysfunctional?
  • Leadership: dictatorship or servanthood?

Why does it matter?

A positive climates increase motivation, innovation and productivity, encourage extra effort – potentially by 30%. Whereas, a negative climate inhibits it (HayGroup).

What is Culture?

In order to change a culture you first need to understand the environment in which it exists.

Often we think of the different cultural groups we participate in as the language we use, the architecture we create, visual arts, literature, music.
These are just manifestations of what culture really is.

Culture does not exist with only one person, individuals exist within a culture.

Culture rules almost all areas of our lives.
The culture is the values behind the behaviours/manifestations of individuals within a culture.
These values are learned.

Why does it matter?

My primary focus in this post is one of providing maximum benefit to our customers.

Getting the best out of our people and putting the best back into our people is secondary. I’ll explain in a section below why our customers should take primary focus and that if they do, most other aspects will fall into place.

Focusing on the Negative biases

Known as Deficit based management, this happens when businesses are tackling their biggest problems in business. I.E. focussing on the negative and how they can remove the problem or reduce it’s effects. Though this technique can be successful in dealing with impediments, removing/reducing areas of poor performance, it does have side effects causing its people to feel overworked and stressed. It produces a general negative attitude and working environment amongst workers. This actually misses some of the largest opportunities to increase the strengths of the business. Because it has us focusing on how we can remove the problems, we miss the opportunities to increase (build on) our strengths.

Focussing on the Positive

What if we focused on our top three customers and which of our strengths have helped to make them successful. Then focus on these strengths and how we can maximise these and broaden the reach of them to effect our other customers. This can help to realign where the organisation is going and bring clarity to what our goals actually are. In a section below I discuss why we shouldn’t focus on the success of our workers but rather the customer.

Organisational Culture Types

Below are the four commonly accepted organisational culture types. The two dimensional view:

Clan

Focusing on Collaboration, how the members can work together in a family-like manner. Focusing on mentoring, nurturing and working together to achieve the result.

Adhocracy

Dynamic and entrepreneurial, focus on taking risks to achieve optimal result. Innovative. Doing things first… driving your designs out with tests. Reactive, ability to move quickly with changing goals. Often appearing as unmanageable chaos. Empirical. Companies like Google embrace this type of culture in which they utilise the skills of entrepreneurial software engineers, cutting edge processes and technologies (Bruce M. Tharp: Haworth).

Hierarchy

Structured and controlled, focusing on efficiency, stability and doing things “the right way”.

Market

Results oriented. Focused on competition, achievement, getting the job done.

The Third Dimension

The third dimension comprises another three organisational culture types. A culture can be created in which it is giving, taking or matching. Attributed to the organisation and/or people within.

Taking

A taking organisation is one where they try to get the best value out of their workers. Workers will often feel used and burnt out. Workers know that they have to work extra hard to prove that they are worth it. Often the workers come to the organisation as takers as well and this helps to solidify the taking culture even more. I’m here to get what I can and then I’ll leave once I have it.

Often have a high staff turnover.

Primary focus: What am I getting, what will my reward look like? It’s all about me.

Matching

Matchers give as much as they take. They stick to the rules. This is one of the attributes of the Hierarchy culture type. They don’t do any extra work unless they’re paid for it. Don’t show much initiative. Parties take account of what they are owed. Workers often stay for a long time, don’t burn out. Don’t innovate or add value to the relationships within the culture.

Primary focus: What am I getting, what will my reward look like? I’m happy to give so long as I get in return.

Giving

A giving culture is one based around serving others. The focus is on how I can make our clients successful.

In a giving culture, a business measures their success by the satisfaction of their clients, rather than on the quantity of effort our employees are giving.
Focus clearly on the value of pleasing the client rather than measuring the value of their effort.
How can we create more value for our clients.

The motivation is targeted at the customer by all parties of the organisation. The consequence (not the focus) is the law of what goes around comes around. You receive what you give.

The organisations that do very well and at the opposite end of the spectrum do very poorly often fall into the same category of givers.

Successful givers work out how the giving will feedback so that they will be enabled to give more, rather than at the other end of the scale where the unsuccessful organisations that give, just keep giving without working out how they can sustain it.

How to change a culture of giving to one of taking or matching

Start rewarding your workers. Provide bonuses and commissions.
If your a giving culture, your focus is on benefiting your customers.
If you start rewarding your workers, their focus changes to look at whether they have the reward rather than the customers.

Often organisations setup reward systems for their employees. One in which the employees are recognised for doing good things. This moves the focus of the organisation from providing benefit to the customers to providing benefit to the employees.

How to change a culture of taking or matching to a giving culture

Stay focused on the value you are providing to your customers.
Focus on the organisations vision of how you’re making the customers lives better.
The mission statement needs to be centred around your customers not your employees or the organisation. Employ people that have the same vision of serving the organisations customers rather than the organisation itself. Don’t reward your workers, but talk about how your workers effected your customers in a positive way.
Don’t tell your customers what they want, you can tell them what they need if they don’t know, because you are the specialist.
Remember you are in business to serve your customers.
The measure of your organisations success should be your clients feedback. Ask your customers what they want. Gather their feedback and insert it into your organisation.
Share the success of your customers rather than your employees. Fix your vision externally rather than looking inward.

Primary focus: What can I give, how can I give.

Effecting Change

Org charts, in difference, don’t show how influence takes place in a business. In reality businesses don’t function through the organizational hierarchy but through its hidden social networks.
People do not resist change or innovation, but they resist the insecurity created by change beyond their influence.
Have you heard the argument that “the quickest way to introduce a new approach is to mandate its use”?
A level of immediate compliance may be achieved, but the commitment won’t necessarily be (Fearless Change 2010).
If you want to bring change, the most effective way is from the bottom up. In saying that, bottom-up takes longer and is harder. Like anything. No pain, no gain. Or as my wife puts it… it’s the difference between instant coffee and espresso.
Top-down change is imposed on people and tries to make change occur quickly and deals with the problems (rejection, rebellion) only if necessary.
Bottom-up change triggered from a personal level focused on first obtaining trust, loyalty, respect (from serving (servant leadership)), and the right to speak (have you served your time, done the hard yards)?

Because the personal relationship and involvement is not usually present with top-down, people will appear to be doing what you mandated, but secretly, still doing things the way they always have done.

The most effective way to bring change is on a local and personal level once you have built good relationships of trust. Anyone can effect change. The most effective change agents are level 5 leaders. These can be found anywhere in an organisation. Not just at the top. Level 5 leaders are:

  1. They are very confident in them selves. Actively seek out successors and enable them to take over.
  2. They are humble, modest and self sacrificing.
  3. They have “unwavering resolve.”
  4. They are work horses rather than show ponies.
  5. They give credit to others for their success and take full responsibility for poor results. They attribute much of their success to ‘good luck’ rather than personal greatness.
  6. They often don’t step forward when a leader is asked for.

Often I’ve thought that if I have an idea I’m sure is better than the existing way of doing things and I can explain logically why it’s better, then people will buy it. All too often this just isn’t the case. People base their decisions on emotions and then justify them with facts.

What I’ve come to realise is that it doesn’t matter how much power or authority you think you have. There is no reason if your able to build a relationship of trust with your peers or even your bosses, that you can not lead them to accept your ideas. The speed at which this may happen is governed by acts, influences and facts such as:

  • Your level of drive tempered with patience
  • The quality of your relationships and the level of trust others have in you
  • To what level do you hold captive their emotions?
  • A genuine appreciation and respect of your people and a belief in them
  • Understanding that people and their acceptance levels are different and how they differ
  • Gentleness
  • Knowing what it means to be a servant leader and being one
  • Mastery of Communication
  • Ability to work well with others
  • A need or problem to be solved
  • Realisation that you shouldn’t attempt to solve everything at once
  • Have you earnt the right to speak (done the hard yards)?
  • The level of support and desire to embrace change that the culture you work within provides
  • The people you want to accept your ideas

This post was leveraged in my talk at AgileNZ 2014. Slide deck here.

Software Engineer Interview Quick Question Set

May 11, 2013

Ice breakers

  • Tell us a little bit about yourself and what drives you?
  • Ask a question from their CV that is positive, ‘what was your greatest success in your current or last role’
  • What’s your ideal job?
  • Can you give us one thing you really enjoyed in your last job?
  • What about one thing that you didn’t enjoy as much?
    How did you solve that?

Testing

  • How can you implement unit testing when there are dependencies between a business layer and a data layer, or the presentation layer and the business layer?
  • The development team is getting near release date. They start saying things like, we’re going to need a sprint to test. What would your reaction be?

Maintenance

  • What measures have you taken to make your software products more easily maintainable?
  • What is the most expensive part of the SDLC?
    (hint: reading others code)

Design and architecture

  • Can you explain some design patterns, and where you have used them?

Scrum

  • Have you used scrum before? (If the answer is no, move on)
  • If you were taken on as a team member and the team was failing Sprint after Sprint. What would you do?
  • What would you do if you were part of a Scrum Team and your manager asked you to do a piece of work not in the Scrum Backlog?
    (hint: manager needs to consult PO. Something has to be removed from Sprint backlog in order for something to be added)

Construction

  • When do you use an abstract class and when do you use an interface?
  • How do you make sure that your code is both safe and fast?
  • Can you describe the process you use for writing a piece of code, from requirements to delivery?

Software engineering questions

  • What are the benefits and drawbacks of Object Orientated Design?
    (hint: polymorphism inheritance encapsulation)
  • What books have you read on software engineering that you thought were good?
  • Explain the terms YAGNI, DRY, SOLID?
    (hint You Aint Gonna Need It. Build what you need as you need it, aggressively refactoring as you go along; don’t spend a lot of time planning for grandiose, unknown future scenarios. Good software can evolve into what it will ultimately become. Every piece of code is code we have to test. If the code is not needed, why are we spending time on it?)

Functional design questions

  • Which controls would you use when a user must select multiple items from a big list, in a minimal amount of space?
  • How would you design editing twenty fields for a list of 10 items? And editing 3 fields for a list of 1000 items?

Specific technical requirements

  • When, where and how do you optimize code?

Web questions

  • How would you mitigate SQL injection?
    (hint: looking for multi layered sanitisation. parameterised SQL. Least privileged account for data access)
  • Have you used XSS and can you provide us an example?
  • What JavaScript libraries have you used?
  • What are some of the irritating limitations of CSS?
  • How would you remove the ASP.NET_SessionId cookie from a MVC controllers Response?
    (hint: Response.Cookies["ASP.NET_SessionId"].Expires = DateTime.Now;)

JavaScript

  • How does JavaScript implement inheritance?
    (hint: via Object’s prototype property)

Service Oriented

  • What are the 3 things a WCF end point must have, or what is the ABC of a WCF service?
    (hint:
    Address – where the WCF service is hosted.
    Binding – that specifies the protocol and its myriad of options.
    Contract – service contract defines what service operations are available to the client for consumption.
    )

C# / .Net questions

  • What’s the difference between public, private, protected and internal modifiers?
  • What are the main differences between the .NET 2.0 and 4.0 garbage collector?
    (hint: background GC was introduced)
  • Describe the different ways arguments can be passed in C#
    (hint: pass val by val, pass val by ref, pass ref by val, pass ref by ref)
  • We have a Base class, we have a child class that inherits BaseClass. Does the child class inherit the base class’s private members?
    (hint: this is normally good for a laugh)
  • Have you ever worked with a deadlock and how did it occur?
  • When should locks be used in concurrent programming?
    (hint:
    when synchronization cannot be performed in any other way. This is rare. With careful thought and planning, there is just about always a better way. There are many ways to synchronise without using locks. System.Threading.Interlocked class generally supported by the processor
    )
  • What are some of your favourite .NET features?

Finally, this question is from Google; can you quickly tell us something that we don’t know anything about? It can be anything.

Software Engineer Interview Process and Questions

April 27, 2013

A short time ago, I was tasked with finding the right software engineer/s for the organisation I was working for. I settled on a process, a set of background questions,  a set of practical programming exercises and a set of verbal questions. Later on I cut the set of verbal questions down to a quicker set. In this post, I’ll be going over the process and the full set of verbal questions. In a subsequent post I’ll go over the quicker set.

The Process

  1. We sent them an email with a series of questions.
    Technical and non-technical.
    They have two days to reply with answers.
    The programming exercises are not covered here.
    If they passed this…
  1. We would get them in for an interview.
    Technical and non-technical questions would be asked.
    They would be put on the spot and asked to speak to the development team about a technical subject that they were familiar with.
    The development team would quiz them on whatever comes to mind.
    Once the candidate had left, the development team would collaborate on what they thought of the candidate and whether or not they would be a good fit for the team.
    The team would take this feedback and discuss whether the candidate should be given a trial. 
    Step 2 could be broken into two parts depending on how many questions and their intensity, you wanted to drill the candidate with.

The following set of tests will confirm whether the candidate satisfies the points we have asked for in the job description.

The non functional (soft) qualities listed on the Job add would need to be kept in mind during the interview events.

Qualities such as:

  • Quality focus
  • Passion
  • Personality
  • Commitment to the organisations needs
  • A genuine sense of excitement about the technologies we work with

Email test

  1. Send Screening.pdf
  2. Send InterviewQuestions.doc

Now with the following questions, with many of them there is not necessarily a right or wrong answer. Many of them are just to gauge how the candidate thinks and whether or not they hold the right set of values.

Ice breakers

  • Would you like to be the team leader or team member?
  • Tell me about a conflict at a previous job and how you resolved it.
  • (Summary personality item: Think to yourself, “If we hire this person, would I want to spend four hours driving in a car with them?”)

Design and architecture

  • What’s the difference between TDD and BDD and why do they matter?
  • What is Technical Debt. How do you deal with it once in it? How do you stay out of it?
  • How would you deal with a pair when reviewing their code, when they have not followed good design principles?
  • What would you do if a fellow team member reviewed your code and suggested you change something you had designed that followed good design principles, to something inferior?
  • Can you explain how the Composite pattern works and where you would use it?
  • Can you describe several class construction techniques?
    What are two design patterns that are focused on class construction, and how do they work?
    (hint: Builder, Factory Method).
  • How would you model the animal kingdom (with species and their behaviour) as a class system?
    (hint GoF design pattern. Abstract Factory)
  • Can you name a number of non-functional (or quality) requirements?
  • What is your advice when a customer wants high performance, high usability and high security?
  • What is your advice when a customer wants high performance, Good design, Cheap?
    (hint: pick 2)
  • What do low coupling and high cohesion mean? What does the principle of encapsulation mean to you?
  • Can you think of some concurrency patterns?
    (hint: Asynchronous Results, Background Worker, Compare/Exchange pattern via Interlocked.CompareExchange)
  • How would you manage conflicts in a web application when different people are editing the same data?
  • Where would you use the Command pattern?
  • Do you know what a stateless business layer is? Where do long-running transactions fit into that picture?
    (hint: if you have long-running transactions, you are going to have to manage state somehow. How would you do this?)
  • What kinds of diagrams have you used in designing parts of an architecture, or a technical design?
  • Can you name the different tiers and responsibilities in an N-tier architecture?
    (hint: presentation, business, data)
  • Can you name different measures to guarantee correctness and robustness of data in an architecture?
    (hint: for example transactions, thread synchronisation)
  • What does the acronym ACID stand for in relation to transactions?
    (hint: atomicity, consistency, isolation, durability)
  • Can you name any differences between object-oriented design and component-based design?
    (hint: objects vs services or documents)
  • How would you model user authorization, user profiles and permissions in a database?(hint: Membership API)

Scrum questions

  • Have you used Scrum before? (If the answer is no, not much point in asking the rest of these questions).
  • If you were taken on as a team member and the team was failing Sprint after Sprint. What would you do?
  • What are the Scrum events and the purpose of them?
    (hint: Daily Scrum, Sprint Planning Meetings 1 & 2, Sprint Review and Sprint Retrospective)
  • What would you do if you were part of a Scrum Team and your manager asked you to do a piece of work not in the Scrum Backlog?
  • Who decides what Product Backlog Items should be pulled into a Sprint?
  • What is the DoD and what is it useful for?
  • Where and how do changing requirements fit into scrum?

Construction questions

  • How do you make sure that your code can handle different kinds of error situations?
    (hint: TDD, BDD, testing…)
  • How do you make sure that your code is both safe and fast?
  • When would you use polymorphism and when would you use delegates?
  • When would you use a class with static members and when would you use a Singleton class?
  • Can you name examples of anticipating changing requirements in your code?
  • Can you describe the process you use for writing a piece of code, from requirements to delivery?
  • Explain DI / IoC. Are there any differences between the two? If so, what are they?
    (hint: DI is one method of following the Dependency Inversion Principle (DIP) or IoC)

Software engineering skills

  • What is Object Oriented Design? What are the benefits and drawbacks?
    (hint: polymorphism inheritance encapsulation)
  • What is the role of interfaces in design?
  • What books have you read on software engineering that you thought were good?
  • What are important aspects of GUI design?
  • What Object Relational Mapping tools have you used?
  • What are the differences between Model-View-Controller, Model-View-Presenter and Model-View-ViewModel
    Can you draw MVC and MVP?
    (hint: doted lines are pub/sub)

MVCM-V-VM

  • What is the difference between Mocks, Stubs, Fakes and Dummies?
  • (hint:
    Mocks are objects pre-programmed with expectations which form a specification of the calls they are expected to receive. Stubs provide canned answers to calls made during the test, usually not responding at all to anything outside what’s programmed in for the test.
    Stubs may also record information about calls, such as an email gateway stub that remembers the messages it ‘sent’, or maybe only how many messages it ‘sent’.
    Fake objects actually have working implementations, but usually take some shortcut which makes them not suitable for production (an in memory database is a good example).
    Dummy objects are passed around but never actually used. Usually they are just used to fill parameter lists.)
  • Describe the process you would take in setting up CI for our company?
  • We’re going to design the new IMDB.
    On the whiteboard, what would the table that holds the movies look like?
    Every movie has actors, how would the Actors table look?
    Actors star in many movies, any adjustments?
    We need to track Characters also. Any adjustments to the schema?

Relational Database

  • What metrics, like cyclomatic complexity, do you think are important to track in code?

Functional design questions

  • What are metaphors used for in functional design? Can you name some successful examples?
    (hint: Partial Function Application, Currying)
  • How can you reduce the user’s perception of waiting when some routines take a long time?
  • Which controls would you use when a user must select multiple items from a big list, in a minimal amount of space?
  • How would you design editing twenty fields for a list of 10 items? And editing 3 fields for a list of 1000 items?
  • Can you name some limitations of a web environment vs. a Windows environment?

Specific technical requirements

  • What software have you used for bug tracking and version control?
  • Which branching models have you used?
    (hint: No Branches, Release, Maintenance, Feature, Team)
  • What have you used for unit testing, integration testing, UA testing, UI testing?
  • What build tools are you familiar with?
    (hint: Nant, Make, Rake, PSake)

Web questions

  • Would you use a black list or white list? Why?
  • Can you explain XSS and how it works?
  • Can you explain CSRF? and how it works?
  • What is the difference between GET and POST in web forms? How do you decide which to use?
  • What do you know about HTTP.
    (hint: Application Layer of OSI model (layer 7), stateless)
  • What are the HTTP methods sometimes called verbs?
    (hint: there are 9 of them. HEAD, GET, POST, PUT, DELETE, TRACE, OPTIONS, CONNECT, PATCH)
  • How do you get the current users name from an MVC Controller?
    (hint: The controller has a User property which is of type IPrinciple which has an Identity property of type IIdentity, which has a Name property)
  • What JavaScript libraries have you used?
  • What is the advantage of using CSS?
  • What are some of the irritating limitations of CSS?

JavaScript questions

  • How does JavaScript implement inheritance?
    (hint: via Object’s prototype property)
  • What is the difference between "==" and "===", "!=" and "!=="?
    (hint: If the two operands are of the same type and have the same value, then “===” produces true and “!==” produces false. The evil twins do the right thing when the operands are of the same type, but if they are of different types, they attempt to coerce the values. The rules by which they do that are complicated and unmemorable.
    If you want to use "==", "!=" be sure you know how it works and test well.
    By default use “===” and “!==“. )
    These are some of the interesting cases:
'' == '0'          // false
0 == ''            // true
0 == '0'           // true
false == 'false'   // false
false == '0'       // true
false == undefined // false
false == null      // false
null == undefined  // true
' \t\r\n ' == 0    // true
  • On the whiteboard, could you show us how to create a function that takes an object and returns a child object?
if (typeof Object.create !== ‘function’) {
   Object.create = function (o) {
      var F = function () {};
      F.prototype = o;
      return new F();
   };
}
var child = Object.create(parent);
  • When is “this” bound to the global object?
    (hint: When the function being invoked is not the property of an object)
  • With the following code, how does myObject.pleaseSetValue set myObject.value?
var myObject = {
	value: 0
};

myObject.setValue = function () {
	var that = this; // don’t show this

	var pleaseSetValue = function () {
		that.value = 10; // don’t show this
	};
	pleaseSetValue ();
}
myObject.setValue();
document.writeln(myObject.value); // 10

Service Oriented questions

  • Can you think of any Advantages and Disadvantages in using SOA over an object oriented n-tier model?
  • What’s the simplest way to make a service call from within a web page and how many lines could you do this in?
  • What scales better, per-call services or per-session and why?
    (hint: maintaining service instances (maintaining state) in memory or any entities for that matter quickly blows out memory and other resources.)
  • What is REST’s primary objective?
  • How many ways can you create a WCF proxy?
    (hint:
    Add Service Reference via Visual Studio project
    Using svcutil.exe
    Create proxy on the fly with… new ChannelFactory<IMyContract>().CreateChannel();
    )
  • What do you need to turn on on the service in order to create a proxy?
    (hint: enable an HTTP-GET behaviour, or MEX endpoint)

C# / .Net questions

  • What’s the difference between public, private, protected and internal modifiers?
    Which ones can be used together?
  • What’s the difference between static and non-static methods?
  • What’s the most obvious difference in IL with static constructors?
    (hint: static method causes compiler to not mark type with beforefieldinit, thus giving lazy initialisation.)
  • How have you used Reflection?
  • What does the garbage collector clean up?
    (hint: managed resources, not unmanaged resources. Such as files, streams and handles)
  • Why would you implement the the IDisposable interface?
    (hint: clean up resources deterministically. Clean up unmanaged resources.)
  • Where should the Dispose function be called from?
    (hint: the objects finalizer)
  • Where is an objects finalizer called from?
    (hint: the GC)
  • If you call an objects Dispose method, what System method should you also make sure is called?
    (hint: System.GC.SuppressFinalize)
  • Why should System.GC.SuppressFinalize be called?
    (hint: finalization is expensive)
  • Are strings mutable or immutable?
    (hint: immutable)
  • What’s the most significant difference between struct’s and class’s?
    (hint: struct : value type, class : reference type)
  • What are the other differences between struct’s and class’s?
    (hint: struct’s don’t support inheritance (all value types are sealed) or finalizers)
    (hint: struct’s can have the same fields, methods, properties and operators)
    (hint: struct’s can implement interfaces)
  • Where are reference types stored? Where are value types stored?
    (hint:
    bit of a trick question. Ref on the heap, val on the stack (generally)
    The reference part of reference type local variables is stored on the stack.
    Value type local variables also on the stack.
    Content of reference type variables is stored on the heap.
    Member variables are stored on the heap.
    )
  • Where is the yield key word used?
    (hint: within an iterator)
  • What are some well known interfaces in the .net library that iterators provide implementation for?
    (hint: IEnumerable<T> )
  • Are static methods thread safe?
    (hint: a new stack frame is created with every method call. All local variables are safe… so long as they are not reference types being passed to another thread or being passed to another thread by ref.)
  • What is the TPL used for?
    (hint: a set of API’s in the System.Threading and System.Threading.Tasks namespaces simplifying the process of adding parallelism and concurrency to applications.)
  • What rules would you consider when choosing a lock object?
    (hint: keep the scope as tight as possible (private), so other threads cannot change its value, thus causing the thread to block.
    Declare as readonly, as its value should not be changed.
    Must not be a value type.
    If the lock keyword is used on a value type, the compiler will report an error.
    If used with System.Threading.Monitor, an exception will occur at runtime, because Monitor.Exit receives a boxed copy of the original variable.
    Never lock on “this”.)
  • Why would you declare a field as volatile?
    (hint: So that the order of the operations performed on the variable are not optimised to a different order.)
  • Are reads and writes to a long (System.Int64) atomic? Are reads and writes to a int (System.Int32) atomic?
    (hint: The runtime guarantees that a type whose size is no bigger than a native integer will not be read or written only partially. This is in the CLI spec and the C# 4.0 spec.)
  • Before invoking a delegate instance just before the null check is performed, What’s a good way to make sure no other threads can set your delegate to null between when the check occurs and when you invoke it?
    (hint:
    assign reference to heap allocated memory to stack allocated implements thread safety.
    Assign your delegate instance to a second local delegate variable.
    This ensures that if subscribers to your delegate instance are removed (by a different thread) between checking for null and firing the invocation, you won’t fire a NullReferenceException.)
void OnCheckChanged(EventArgs e) {
	// assign reference to heap allocated memory to
	// stack allocated implements thread safety

	// CheckChanged is a member declared as…  public event EventHandler CheckChanged;
	EventHandler threadSafeCheckChanged = CheckChanged;
	if (threadSafeCheckChanged != null)  {
		// fire the event off
		foreach(EventHandler handler in threadSafeCheckChanged.GetInvocationList()) {
			try {
				handler(this, e);
			}
			catch(Exception e) {
				// handling code
			}
		}
	}
}
  • What is a deadlock and how does one occur? Can you draw it on the white board?
    (hint: two or more threads wait for each other to release a synchronization lock.
    Example:
    Thread A requests a lock on _sync1, and then later requests a lock on _sync2 before releasing the lock on _sync1.
    At the same time,
    Thread B requests a lock on _sync2, followed by a lock on _sync1, before releasing the lock on _sync2.
    )
  • How many ways are there to implement an interface member, and what are they?
    (hint: two. Implicit and explicit member implementation)
  • How do I declare an explicit interface member?
    (hint: prefix the member name with the interface name)
public class MyClass : SomeBaseClass ,IListable, IComparable {
    // …
    public intCompareTo(object obj) {
        // …
    }

    #region IListable Members
    string[] Ilistable.ColumnValues {

        get {
            // …
            return values;
        }
    }
    #endregion
}
  • Write the above on a white board, then ask the following question. If I want to make a call to an explicit member implementation like the above, How do I do it?
string[] values;
    MyClass obj1, obj2;

    // ERROR:  Unable to call ColumnValues() directly on a contact
    // values = obj1.ColumnValues;

    // First cast to IListable.
    values = ((IListable)obj2).ColumnValues;
  • What is wrong with the following snippet?
    (hint: possibility of race condition.
    If two threads in the program both call GetNext simultaneously, two threads might be given the same number. The reason is that _curr++ compiles into three separate steps:
    1. Read the current value from the shared _curr variable into a processor register.
    2. Increment that register.
    3. Write the register value back to the shared _curr variable.
    Two threads executing this same sequence can both read the same value from _curr locally (say, 42), increment it (to, say, 43), and publish the same resulting value. GetNext thus returns the same number for both threads, breaking the algorithm. Although the simple statement _curr++ appears to be atomic, this couldn’t be further from the truth.)
// Each call to GetNext should hand out a new unique number
static class Counter {
    internal static int _curr = 0;
    internal static int GetNext() {
        return _curr++;
    }
}
  • What are some of your favourite .NET features?

Data structures

  • How would you implement the structure of the London underground in a computer’s memory?
    (hint: how about a graph. The set of vertices would represent the stations. The edges connecting them would be the tracks)
  • How would you store the value of a colour in a database, as efficiently as possible?
    (hint: assuming we are measuring efficiency in size and not retrieval or storage speed, and the colour is 16^6 (FFFFFF), store it as an int)
  • What is the difference between a queue and a stack?
  • What is the difference between storing data on the heap vs. on the stack?
  • What is the number 21 in binary format? And in hex?
    (hint: 10101, 15)
  • What is the last thing you learned about data structures from a book, magazine or web site?
  • Can you name some different text file formats for storing unicode characters?
  • How would you store a vector in N dimensions in a datatable?

Algorithms

  • What type of language do you prefer for writing complex algorithms?
  • How do you find out if a number is a power of 2? And how do you know if it is an odd number?
  • How do you find the middle item in a linked list?
  • How would you change the format of all the phone numbers in 10,000 static html web pages?
  • Can you name an example of a recursive solution that you created?
  • Which is faster: finding an item in a hashtable or in a sorted list?
  • What is the last thing you learned about algorithms from a book, magazine or web site?
  • How would you write a function to reverse a string? And can you do that without a temporary string?
  • In an array with integers between 1 and 1,000,000 one value is in the array twice. How do you determine which one?
  • Do you know about the Traveling Salesman Problem?

Testing questions

  • It’s Monday and we’ve just finished Sprint Planning. How would you organize testing?
  • How do you verify that new changes have not broken existing features?
    (hint: regression test)
  • What can you do reduce the chance that a customer finds things that he doesn’t like during acceptance testing?
  • Can you tell me something that you have learned about testing and quality assurance in the last year?
  • What sort of information would you not want to be revealed via Http responses or error messages?
    (hint: Critical info about the likes of server name, version, installed program versions, etc)
  • What would you make sure you turned off on an app or web server before deployment?
    (hint: directory listing?)

Maintenance questions

  • How do you find an error in a large file with code that you cannot step through?
  • How can you make sure that changes in code will not affect any other parts of the product?
  • How can you debug a system in a production environment, while it is being used?

Configuration management questions

  • Which items do you normally place under version control?
  • How would you manage changes to technical documentation, like the architecture of a product?

Project management

  • How many of the three variables scope, time and cost can be fixed by the customer?
  • Who should make estimates for the effort of a project? Who is allowed to set the deadline?
  • Which kind of diagrams do you use to track progress in a project?
  • What is the difference between an iteration and an increment?
  • Can you explain the practice of risk management? How should risks be managed?
  • What do you need to be able to determine if a project is on time and within budget?
    (hint: Product Backlog burn-down)
  • How do you agree on scope and time with the customer, when the customer wants too much?

Candidate displays how they communicate / present to a group of people about a technical topic they are passionate and familiar about.

References I used

If any of these questions or answers are not clear, or you have other great ideas for questions, please leave comments.

Running Wireshark as non-root user

April 13, 2013

As part of my journey with Node.js I decided I wanted to see exactly what was happening on the wire. I decided to use Burp Suite as the Http proxy interceptor and Wireshark as the network sniffer (not an interceptor). Wireshark can’t alter the traffic, it can’t decrypt SSL traffic unless the encryption key can be provided and Wireshark is compiled against GnuTLS.

This post is targeted at getting Wireshark running on Linux. If you’re a windows user, you can check out the Windows notes here.

When you first install Wireshark and try to start capturing packets, you will probably notice the error “You didn’t specify an interface on which to capture packets.”

When you try to specify an interface from which to capture, you will probably notice the error “There are no interfaces on which a capture can be done.”

You can try running Wireshark as root: gksudo wireshark

Wireshark as root

This will work, but of course it’s not a good idea to run a comprehensive tool like Wireshark (over 1’500’000 lines of code) as root.

So what’s actually happening here?

We have dumpcap and we have wireshark. dumpcap is the executable responsible for the low level data capture of your network interface. wireshark uses dumpcap. Dumpcap needs to run as root, wireshark does not need to run as root because it has Privilege Separation.

If you look at the above suggested “better way” here, this will make a “little” more sense. In order for it to make quite a lot more sense, I’ll share what I’ve just learnt.

Wireshark has implemented Privilege Separation which means that the Wireshark GUI (or the tshark CLI) can run as a normal user while the dumpcap capture utility runs as root. Why can’t this just work out of the box? Well there is a discussion here on that. It doesn’t appear to be resolved yet. Personally I don’t think that anybody wanting to use wireshark should have to learn all these intricacies to “just use it”. As the speed of development gets faster, we just don’t have time to learn everything. Although on the other hand, a little understanding of what’s actually happening under the covers can help in more ways than one. Anyway, enough ranting.

How do we get this to all “just work”

from your console:

sudo dpkg-reconfigure wireshark-common

You’ll be prompted:

Configuring wireshark-common

Respond yes.

The wireshark group will be added

If the Linux Filesystem Capabilities are not present at the time of installing wireshark-common (Debian GNU/kFreeBSD, Debian GNU/Hurd), the installer will fall back to set the set-user-id bit to allow non-root users to capture packets. Custom built kernels may lack Linux Capabilities.

The help text also warns about a security risk which isn’t an issue because setuid isn’t used. Rather what actually happens is the following:

addgroup --quiet --system wireshark
chown root:wireshark /usr/bin/dumpcap
setcap cap_net_raw,cap_net_admin=eip /usr/bin/dumpcap

You will then have to manually add your user to the wireshark group.

sudo adduser kim wireshark # replacing kim with your user

or

usermod -a -G wireshark kim # replacing kim with your user

log out then back in again.

I wanted to make sure that what I thought was happening was actually happening. You’ll notice that if you run the following before and after the reconfigure:

ls -liah /usr/bin/dumpcap | less

You’ll see:

-rwxr-xr-x root root /usr/bin/dumpcap initially
-rwxr-xr-x root wireshark /usr/bin/dumpcap after

And a before and after of my users and groups I ran:

cat /etc/passwd | cut -d: -f1
cat /etc/group | cut -d: -f1

Alternatively to using the following as shown above, which gives us a nice abstraction (if that’s what you like):

sudo dpkg-reconfigure wireshark-common

We could just run the following:

addgroup wireshark
sudo chgrp wireshark /usr/bin/dumpcap
sudo chmod 750 /usr/bin/dumpcap
sudo setcap cap_net_raw,cap_net_admin+eip /usr/bin/dumpcap

The following will confirm the capabilities you just set.

getcap /usr/bin/dumpcap

What’s with the setcap?

For full details, run:

man setcap
man capabilities

setcap sets the capabilities of each specified filename to the capabilities specified (thank you man ;-))

For sniffing we need two of the capabilities listed in the capabilities man page.

  1. CAP_NET_ADMIN Perform various network-related operations (e.g., setting privileged socket options, enabling multicasting, interface configuration, modifying routing tables). This allows dumpcap to set interfaces to promiscuous mode.
  2. CAP_NET_RAW Use RAW and PACKET sockets. Gives dumpcap raw access to an interface.

For further details check out Jeremy Stretch’s explanation on Linux Filesystem Capabilities and using setcap. There’s also some more info covering the “eip” in point 2 here and the following section.

man capabilities | grep -A24 "File Capabilities"

Lets run Wireshark as our usual low privilege user

Now that you’ve done the above steps including the log off/on, you should be able to run wireshark as your usual user and configure your listening interfaces and start capturing packets.

Also before we forget… Ensure Wireshark works only from root and from a user in the “wireshark” group. You can add a temp user (command shown above).

Log in as them and try running wireshark. You should have the same issues as you had initially. Remove the tempuser:

userdel -r tempuser

Setup of Chromium, Burp Suite, Node.js to view HTTP on the wire

March 30, 2013

As part of my Node.js development I really wanted to see what was going over the wire from chromium-browser to my Node.js web apps.

I have node.js installed globaly, express installed locally, a very simple express server listening on port 3000

var express = require('express');
var app = express();

app.get('/', function (request, response) {
   response.send('Welcome to Express!');
});

app.listen(3000);

Burp Suite setup in my main menu. Added the command via System menu -> Preferences -> Main Menu

Burp Suite Command

The Command string looks like the following.

java -jar -Xmx1024m /WhereTheBurpSuiteLives/burpsuite_free_v1.5.jar

Setting up Burp Suite configuration details are found here. I’ve used Burp Suite before several times. Most notably to create my PowerOffUPSGuests library which I discuss here. In that usage I reverse engineered how the VMware vSphere client shuts down it’s guests and replicated the traffic in my library code. For a simple setup, it’s very easy to use. You can spend hours exploring Burps options and all the devious things you can use it for, but to get started it’s simple. Set it up to listen on localhost and port 3001 for this example.

Burp Suite Proxy Listeners

Run the web app

to start our express app from the directory where our above server is located, from a console, run:

node index.js

Where index.js is the name of the file that contains our JavaScript.

To test that our express server is active. We can browse to http://localhost:3000/ or we can curl it:

curl -i  http://localhost:3000/

Should give us something in return like:


HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: text/html; charset=utf-8
Content-Length: 19
Date: Sun, 24 Mar 2013 07:53:38 GMT
Connection: keep-alive

Welcome to Express!

Now for the Proxy interception (Burp Suite)

Now that we’ve got end to end comms, lets test the interceptor.

Run burpsuite with the command I showed you above.

Fire the Http request at your web app via the proxy:

curl -i --proxy http://localhost:3001 http://localhost:3000/

Now you should see burps interceptor catch the request. On the Intercept tab, press the Forward button and curl should show a similar response to above.

Burp Suite Proxy Intercept

If you look at the History tab, you can select the message curl sent and also see the same Response that curl received.

Burp Suite Proxy History

Now you can also set Burp to intercept the server responses too. In fact Burp is extremely configurable. You can also pass the messages to different components of Burp to process how ever you see fit. As you can see in the above image looking at all the tabs that represent burp tools. These can be very useful for penetration testing your app as you develop it.

I wanted to be able to use chromium normally and also be able to open another window for browsing my express apps and viewing the HTTP via Burp Suite. This is actually quite simple. Again with your app running locally on port 3000 and burp listening on port 3001, run:

chromium-browser --temp-profile --proxy-server=localhost:3001

For more chromium options:

chromium-browser -help

Now you can just browse to your web app and have burp intercept your requests.

chromium proxied via burp

You may also want to ignore requests to your search provider, because as your typing in URL’s chromium will send searches when you pause. Under Proxy->Options tab you can do something like this:

Ignore Client Requests