Software Development (or Engineering), however, is not.  The ACM has a special interest group dedicated to the study of Software Engineering but from what I have seen most the research that is conducted under this group is, to put it bluntly, irreveleant.  The problem is that software engineering can’t be isolated like a ray tracing algorithm can.  Software engineering is about people and processes and what works and what doesn’t work – none of which are easy to measure or test.

I am currently attempting to write a paper on Test-Driven Development and Behaviour-Driven development.  It is for my undergraduate senior seminar (a capstone class – kind of) and is supposed to be based on peer-reviewed research.  The problem is that the peer-reviewed research is lagging behind practice by several years.  There has been some scholarly research done on test-first software development (TDD) but not too much.  There has been some research done on specific aspects of software testing but again, not too much.

As easy as it would be to say “the academic institution has failed”, I can’t. I understand why little research has been done on TDD and why none has been done on BDD. It is hard.

Imagine that you are a researcher who wants to see if BDD is “better than” TDD.  First define “better than” as reducing the number of defects in a released software system and reducing the amount of time to design and implement a software system.  To get a point of reference we will need to be able to measure how large a “software system” is.  While hard, this is doable software engineers have come up with all sorts of metrics – pick one (or an aggregate of a couple) and stick with it.  Next we need to find huge group of software development teams – some need to be using TDD and some need to be using BDD.  Before they all start projects we have to have them (or us) measure the size of the projects that they are about to start.  We need to judge their compitency using either TDD or BDD.  Then we have to follow all the teams throughout the development process observing how things go.  After the software is released we can measure how much time each project took.  We also need to start tracking software defects as the life of each software system goes on.

Once we gather all this data we can do some math and statistics and figure out which practice is better. Right?  Well… maybe.  The first issue is that in order for there to be any confidence in this study we would need to study hundereds if not thousands of software development teams in order to balance out the different sizes of projects, the different environments that they systems are being developed in, and the skill levels of the different teams. Then you have to take other things into consideration: Is there a bias against BDD since it is a newer methodology and the teams using it have less experience with it?  Is there a bias against TDD because more advanced and experienced teams will move to BDD quicker? Like I said, it is hard.  Measuring something as simple seeming as “is TDD or BDD more effective?” turns out to be an amazingly large undertaking.

There doesn’t seem to be an easy answer to this – hopefully academic research in the area of software engineering will progress but only time will tell.