How To Write A Good Programming Test
I’ve taken and marked a lot of programming tests in the past. I love doing them as it’s always good to see what challenges different programmers have come up with when designing them. Unfortunately, however, most of them aren’t very good. Below is a list of general ideas to help increase the quality of programmer tests.
1. Keep it relevant
Too many of the tests have questions on content that simply isn’t relevant to the job. The whole idea behind a programming test is making sure the candidate is competent enough to fulfill the role. It shouldn’t be asking questions about arbitrary knowledge that you think ‘good programmers’ should have. That’s not to say that you shouldn’t be testing the basics, as your programmers will need to know these. The important thing is that you don’t test certain concepts just for the sake of it.
A good example of this is “friends” in C++. Nearly every programming test I’ve taken has asked a question about these. I’ve been programming for 5 years now, and I’ve only used them twice. Firstly at university and secondly when taking programming tests. I’ve asked a few other senior programmers and none of them have used them either. Why bother asking a question if it’s not relevant to the work they’ll be doing? I’m sure some people somewhere use them, and if that’s important, ask away, however if most programmers on your team have never touched them, why is it important that new programmers know them? Chances are, if the new programmer does use them, no one else on the team will understand them, or have a different solution to solving the same problem.
Of course, it would be ideal to hire programmers that just know everything there is to know about programming. In reality this isn’t going to be the case. It’s therefore important to make sure their knowledge domain includes exactly what you need. Asking irrelevant questions won’t help you find better programmers, it will help you find people who know how to pass programming tests.
2. Don’t ask stupid questions
Why bother asking questions about simple compile errors in code that is written on paper. These days, compilers will pick this up far quicker than a person can. Seeing if a programmer can find these simple mistakes is like asking an English professor to find spelling mistakes rather than fixing your sentence structure. Ask programmers to figure out logic errors in code. Get them to follow code logic and determine problems or simple output.
Also, don’t ask programmers trick questions. A lot of the time with trick questions you’ll spend a long time figuring them out, and once you know them, you’ll know instantly. Trick questions generally test whether someone has been asked that question before rather than whether they are a good programmer.
3. Be careful when asking optimization questions
Compilers and platforms are changing extremely quickly. This is particularly true within the games industry where we have entirely new hardware every few years. For these reasons you must be very careful when asking programmers to optimize code. What used to be a smart optimization even a few years ago can often be the slower option now.
For example some modern CPU’s (particularly on consoles) are very slow at executing branches (eg if statements). The simple question of optimizing a routine by using a lookup table that is cached may not necessarily be the fastest optimization anymore. What’s more important is asking people the general concepts of optimization. If they mention profiling with some explanation of best practices then you’re onto a winner.
If your work isn’t specifically about optimization then I would leave out any specific questions altogether. In games, optimized code is very important, so it’s necessary, but again, ask more high-level concepts rather than giving specific code blocks and asking them to write the fastest code. The best candidate would simply say that without knowing the exact hardware and without profiling they would rather write clean code than what they think would be optimized. Code Complete has an excellent section on optimizing code.
4. Watch your team coding, then write the test
If you’re stuck for questions then a practical hint is to simply sit for a little while and watch or talk to your team about what tasks they are performing. Ask the team what they spend most of their time on. You can then write your test around what they spend most of their time doing.
This will have the added bonus of keeping it relevant and stop you from asking optimization questions if they are not needed. I find that I spend 80% of my job doing fairly similar tasks and capturing what skills are used for these tasks and making sure new employee’s have them will almost guarantee you will find a good candidate. Obviously if you’re looking for a specific role to be filled this won’t work, but when looking for general programmers this will help.
5. KISS (Keep it simple stupid)
I’ve found that too many tests have the problem of being too complex. You know exactly what question they are asking, however they use a large convoluted example to ask the question, rather than simply asking the question itself. Rather than having a large “const” example and having the tester take minutes to figure things out, why not ask them specifically what “const” means.
The questions shouldn’t be complex, the answer should. While you don’t just want textbook answers, it’s better to keep anything complex to an interview. This leads to my final tip.
6. Use the interview for determining thought process
Programming tests are great for weeding out the programmers that just can’t program (and unfortunately there are a lot of them). They aren’t good at finding out the thought processes the programmer has when solving problems. Trying to find this out from a written test just won’t work, leave it for the interview.
I like to ask simple questions where the programmer has to solve a coding problem on a whiteboard. Rather than finding out whether they can write code on paper, give them something that doesn’t use complex code, but that requires a complex design. The main aim of this is to find out the thought processes when solving problems. As a programmer, you’ll know the “right” way to solving the problem and you’ll soon tell if the programmer has no idea what they are doing. You can also be surprised by an interesting solution that you may not have thought of. Either way, don’t be critical of the code they write, but rather of the way they go about solving it. Tell them to say out loud what they are thinking as they solve the problem. Think about how those thought processes may extend to larger problems and whether you want that type of person on your team.
Conclusion
There are a few simple tips for writing a good programming test. I’d love to see more relevant tests that actually find good programmers rather than the cookie cut list of questions that try and find a specific type of programmer (whether good or not).
If you have any questions you’ve particularly liked or disliked in tests in the past please add them as a comment so we can discuss, and perhaps come up with a better question that looks for the same answer.
So, in regards to your “friends” in #1, I think that’s actually a really useful feature. An example, if you’re writing an engine, you can have “private” functions inside the engine classes by marking the other classes as friends.
I don’t disagree with your point, just your example
#2 I do disagree with, especially for projects which have massive build times…like 20+ minutes, then its great to have coders that know what compiling code looks like
But yea, the trick questions / trivia about the language is usually worthless, but it can indicate how much time the canidate has put into learning the language.
#3,4,5 I think its a great points, especially #4 thanks for the tip!
#6 – That’s what the test is supposed to show anyways right? How the candidate thinks, and I think your suggestions of using a white-board and having the canidate think out loud are spot on.
Nice post!
RohoMech-
Thanks for you thoughts.
#1 – I’d like to be proven incorrect on this. It just seems strange that most people I know haven’t used friends before. If you do use friends, are they one of the most important things you use? If not, would it be better to use a different subject to focus questions on?
#2 – I totally agree that you need programmers to understand code, and that they shouldn’t always rely on the compiler. Having said that, I don’t think that making a programmer sift through 10-50 lines of code to find an error is the best way to find out if they are a good programmer.
#6 – The main point here is that it’s far easier to figure out thought processes in an interview rather than simply what their answer is to a test. You can certainly make assumptions about their thought process based on an answer, however trying to gain too much insight into this through a test can be an issue.
Doolwind – np, always happy to contribute to a good post.
#1 – Marking things as friend is sorta important, our libraries would expose a lot of functions then. As a matter of fact, C# even the notion of Friend Assemblies (which is fantastic for writing Unit tests).
But, as you’re basing this post of your experience, we’re just going to be arguing in circles so I’ll concede the point.
#6 – In terms of this, some of my interviews had lengthy exams. But few of them actually had me explain my answers. Explaining what I wrote imo is a great interview approach, because it mirrors how I’ll be coding at work. I’ll probably be given a task, complete it, and report about how things were done. But once again, this can vary greatly from job to job, so I’ll concede this as well.
RohoMech-
Thanks again for your thoughts. You don’t need to concede your points though
.
I really like to have someone explain the way they are solving a problem, and I enjoy doing it just for myself. It can be quite interesting to see how different approaches can come to the same conclusion/code.
“I’ll probably be given a task, complete it, and report about how things were done.”
That’s actually a really good idea. I think that reporting how tasks were completed to other members (particularly senior ones) is a good way of making sure the problems are solved in their most optimal way. This also encourages the seniors to share some of their knowledge if they’ve solved the same problem slightly differently.
I have to agree with the author about Friend classes. The only time I’ve used Friends in 15 years is when using a specific form of unit testing where each object had a friended object that performed tests on it. Topics like this are just never used until someone insists it be done a certain way, and by then I figure I can research it.
If you expecting candidates to be experts on aspects of a language that they never use, then you’re looking for people who spend a lot of their research time on things that aren’t usually valuable and you can expect them to spend their programming time the same way.
I’d like to toss in another bit about trick questions. There are many things that are called “trick questions”, including those for which there are no right answer (e.g. “what’s your worst trait?”). I think that what he’s talking about are the questions for which the answer is either completely non-intuitive, or for which the intuitive answer is the wrong one.
Of the three, the first is somewhat useful because it can tell you how many interviews the person’s been through. If they have good answers for numerous of that type, then they’ve been doing a lot of interviewing. Not that this is meaningful, but it does tell you something. Also, you get the occasional idiot that lets slip that they have their dead mother in their basement.
The second type is sometimes useful to see how people approach a difficult or insoluable problem, or again to see if they interview a lot or regularly read Mensa magazines. Lots of time for little benefit, as the author says.
The third type should always be avoided because they’re usually presented as a way of allowing the interviewer to seem smarter than the interviewee. While we’re at it, any problem presented to the interviewee is bad if it’s geared towards making them look or feel stupid. You don’t want to create a combative relationship with all of your employees before they get in the door.
Good job, Doolwind.
In my upcoming program (42000 loc), I use friend a few times in the backend, for not exposing too much functionality to the frontend. Friends is really useful with the pimpl ideom.
prompt> grep friend *
facade.h: friend class Design;
facade.h: friend class DesignData;
facade.h: friend class FacadeData;
facade.h: friend class Facade;
facade.h: friend class BrickData;
facade.h: friend class Brick;
facade.h: friend class Saver;
facade.h: friend class BrickData;
facade.h: friend class Design;
facade.h: friend class DesignData;
facade.h: friend class Facade;
facade.h: friend class ParameterData;
facade.h: friend class FileIO2;
facade.h: friend class Saver;
facade.h: friend class Loader;
facade.h: friend class UpdateParameterEvent;
facade.h: friend class ColorParameter;
facade.h: friend class FloatParameter;
facade.h: friend class InfileParameter;
facade.h: friend class IntegerParameter;
facade.h: friend class ItemParameter;
facade.h: friend class StringParameter;
intern.h: friend class BrickData;
intern.h: friend class BrickData;
intern.h: friend class BrickData;
intern.h: friend class BrickData;
intern.h: friend class BrickData;
intern.h: friend class BrickData;
intern.h: friend class BrickData;
parameter.h: friend class Brick;
parameter.h: friend class BrickData;
parameter.h: friend class Saver;
parameter.h: friend class ParamToXML;
parameter.h: friend class XMLToParam;
parameter.h: friend class ParamDecode;
parameter.h: friend class Design;
parameter.h: friend class ParameterData;
parameter.h: friend class Brick;
parameter.h: friend class BrickData;
parameter.h: friend class XMLToParam;
parameter.h: friend class ParamDecode;
parameter.h: friend class FloatParameterData;
parameter.h: friend class Brick;
parameter.h: friend class BrickData;
parameter.h: friend class XMLToParam;
parameter.h: friend class ParamDecode;
parameter.h: friend class IntegerParameterData;
parameter.h: friend class Brick;
parameter.h: friend class BrickData;
parameter.h: friend class XMLToParam;
parameter.h: friend class ParamDecode;
parameter.h: friend class ParamEncode;
parameter.h: friend class ItemParameterData;
parameter.h: friend class Brick;
parameter.h: friend class BrickData;
parameter.h: friend class XMLToParam;
parameter.h: friend class ParamDecode;
parameter.h: friend class ColorParameterData;
parameter.h: friend class Brick;
parameter.h: friend class BrickData;
parameter.h: friend class XMLToParam;
parameter.h: friend class ParamDecode;
parameter.h: friend class InfileParameterData;
parameter.h: friend class Brick;
parameter.h: friend class BrickData;
parameter.h: friend class XMLToParam;
parameter.h: friend class ParamDecode;
parameter.h: friend class StringParameterData;
prompt>
People have different skills, some has good understanding of the language, others are architects, others has collaboration skills, and yet other are multitalented. Tests for a specific thing may rule out good people.
I was looking for a good programming test to save my time not inviting programmers with poor programming skills to the interview. Does anybody knows a PHP similar to interactive Java coding test at http://www.betterprogrammer.com ?
Hey guys, Alistair is right. In most of cases using friend relationships between classes is a sign of bad design.
Of course by some exceptions like Factory pattern etc.