Posted by: jsonmez | December 22, 2010

Solving Problems, You Better Learn How

It is astounding how many developers can write and maintain large enterprise systems dealing with all kinds of complex logic, database access, etc and cannot for the life of them solve a moderately difficult programming problem given in an interview in less than 30 minutes.

It is also astounding how many developers that can not write even a single line of code can fail the same problem exactly the same way.

Based on those two statements, your first assumption might be that asking someone to solve a programming problem as part of an interview would be a bad idea, because it isn’t really telling you if they are good and just crumble when asked to do something like this on the spot, or if they can’t program at all.


Let’s look at some possible numbers to see why this is wrong

In poker you can use a combination of pot odds and the likely holdings of your opponents to deduce in many situations whether or not you should call a bet.

Pot odds is simply a ratio of how much money is in the pot (what you can win), versus how much it will cost you to call the bet.

For example, if a pot contained $9 and I bet you $1, you would be getting 10:1 odds on your money for a call (you have to risk an additional dollar to potentially win 10.)

So in that example, if you have a better than 10% chance of winning, you are making a profitable play by calling.

I don’t have the exact numbers (actually I am just guessing at them from experience), but here is a realistic estimate:

  • Good programmers – 60% can solve the problems
  • Great programmers – 90% can solve the problems
  • Don’t know how to program, lied on my resume – 1% can solve the problems by some sort of divination or black magic, or memorizing sequences of symbols which solve the problem.

Given some statistics like this, if you have someone pass this kind of a test, you can calculate the rough probability of which of the three groups they belong to.

  • 40% chance they are Good
  • 59.5% chance they are Great
  • .5% chance they are a black magic practicing witch or warlock
  • 99.5% chance they are either good or great

So, even though you may be throwing away some good candidates and some great ones, the chance of you getting someone who doesn’t know how to code at all is almost reduced to nothing.  This is very important!

Why?  Because if you hire someone who doesn’t know how to code at all, it will cost you a huge amount of money, but if you pass over someone who is great, but still end up getting someone else who is good or great it doesn’t really cost you anything except in the rare case that the great programmer failed out of the test and you ended up hiring a good one instead.

Even still, that is a small loss versus hiring someone who can’t write a lick of code.

I’ve talked about how I think having code in an interview is a good thing before, but now the numbers are here to back me up.

More and companies are realizing the truth about this situation and companies like Codility are popping up to offer a service to test candidates for you.

What this means for you?

You might not get asked to solve a problem or write code in your next interview, but there is a growing chance that you will.

You may think to yourself, ok even if I bomb the code writing part, I will do great on the rest of the interview and be able to convince them that I know how to write code, besides any company that would toss me out of the running for failing some stupid online test is a company I don’t want to work for anyway.

To that I say a company that doesn’t know how to do math is a company that you don’t want to work for. If my numbers are anything close to accurate, the mathematical best choice is to almost always throw out someone who fails the coding test.

The simple solution is to learn how to solve these kinds of problems.  Learn how to write an algorithm under pressure in 30 minutes.  It’s not that hard to practice and the skill will help you in many other areas of your career and probably your life in general.

I’ve written before on TopCoder as a great site to practice these kinds of problems, and if you haven’t checked it out that is a great place to start.

Another option is to get the book Programming Pearls.  It is a bit dated, but has some nice programming problems for you to try to solve.

I’ll follow up this post in the coming weeks with some hints and tips on how to take a programming problem, break it down and finally code it.  Learning how to do this properly can make some of the more difficult algorithm type problems very simple.

So get to it!  Start learning how to solve programming problems.  Reverse strings, sort linked lists, help grandma pick the highest yielding grandsons and a granddaughters to send Christmas cards to!

If you are following my back to basics series, don’t worry, it is still on.  I’ll probably pick back up with it after the holidays.

As always, you can subscribe to this RSS feed to follow my posts on Making the Complex Simple.  Feel free to check out where I post about the topic of writing elegant code about once a week.  Also, you can follow me on twitter here.


  1. Forgot to mention… Hiring managers, if you know someone who you have already seen that they can write good production quality code, having them take a test could lead you in the wrong direction about that person.
    The whole idea of the mathematics above is that you don’t know if that person can actually code or if they are any good. If you already know information about a developer, (you have seen their code or you have heard good things about code they write etc), then if they happen to not work well under a time deadline or extreme pressure, you are going to think they are not good.

    Point is, this is NOT, I repeat a NOT good way to evaluate someone who you already know a good deal of information about. Ask their peers or those who report to them, or look at their work.

    A code test is good for weeding out bad candidates by being willing to toss out some good and greats ones to make sure you don’t get the really bad ones to slip through and BS you.

  2. Well your math is all wrong. To get the conclusion you got you need to assume that the world of candidates is divided into three equally large groups – 33.3% Great developer, 33.3% Good developer, 33.3% can’t develop.

    I’d say the candidate world is more like 2% great developer, 23% good developer, 75% not good enough.

    And the not good enough have a slightly better percentage to win. I’d say about 5%.
    This means that if someone passes your test, he has:
    9.3% chance of being great
    71.3% chance of being good
    19.3% chance of being not good enough.
    (Even if you take 1% chance of them succeeding, a passing candidate has a 4.6% chance of being a diviner)

    So he’s still likely to be good. But you’ve also interpreted the results wrong.
    “the mathematical best choice is to almost always throw out someone who fails the coding test”

    Let’s see what happens if they fail.

    0.2% chance of being great
    11.4% of being good
    88.3% of being not good enough
    So there’s still more than 10% chance to that he is good.

    (In your world of 1% pass for diviners and 33% for each group the chances are even better for a good developer – 0.6%/27%/66%)

    Now consider my company, where we don’t want good developers – we only want the great ones.

    If you fail, you’re very likely not a great developer – 99.8%. If you pass, you have a 90.7% chance of not being great either – this test is obviously not a good enough metric.
    Which is why we don’t have a 30-minute test. We have more than one test, and the grade for each is not pass/fail. How you think in the test is far more important than the quality of your original answer.

    Another thing I want to point out about a company that only wants great developers: “if you pass over someone who is great, but still end up getting someone else who is good or great” – this is simply not true for us. We can’t afford to pass over someone who is great, because he is also irreplaceable. Which is another reason our testing is a big longer than others – if we’re unsure at any test before the last one, you’ll pass to the next test. Only if we’re sure there’s no chance you’re good enough for us will we let you go.

    • I mostly agree with what you said. Very good points.
      A couple of points though, so as not to detract from what I said above, because clearly it doesn’t apply to your company.

      1. You are right about my math. I went with the assumption, that an even number of great, goods, and craps walked into the interview. It’s a bit skewed, but not that much. When I say great, I mean better than average, and my good is average or slightly above.

      2. To find the cream of the crop, which it sounds like your company is looking for it is going to take a large amount of work and a simple test will not do.

      3. I am talking about a screening test here. Use it to screen, not to evaluate.

      4. Most companies aren’t going to spend the time or energy required to get the top 1-2% candidates. I wish more companies would. It is worth the effort and money in my opinion, but reality is they won’t. What I am suggesting is what most companies will probably end up doing.

      Excellent points though, and solid math based your estimated distribution, which I agree with. Always glad to hear a reply from you, because you always add good content.

      • “I wish more companies would.”

        I don’t; leave the good developers for the good companies, I say!

  3. […] for the life of them solve a moderately difficult programming problem given in an interview in… [full post] jsonmez Making the Complex Simple mathalgorithmsinterviewcareer 0 0 […]

  4. May I ask, which kind of problem do you consider “moderately difficult”, and which is hard?
    Would it be ok for a candidate to write pseudocode, or you prefer actual implementation in a programming language?
    I’m asking this cause just today I was asked to write a multithreading application with rollback capabilities to get all the source code of the pages contained in a website (e.g. Not so difficult using pseudocode (but not even simple, since it includes use of recursion, text parsing, and multithreading), but for a not expert developer (less than 2 years experience), but quite hard to write it with existing java functions, on a dashboard without google, in my opinion. Or am I wrong? How do you consider a problem like this?

  5. […] Right before the holidays, I said that you had better learn how to solve programming problems. […]

  6. You said that you will give some hints and tips on how to take a programming problem, break it down and finally code it. Where it is?

  7. The numbers in this post are completely arbitrary.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: