You've done a good job of pointing out weaknesses in the paper, but I have a question about your defect rate argument: Would you hold the same position if the study had said the opposite? In other words, if it claimed that Ruby and Perl had more bug fixes (and thus defects) than Erlang and Java, would you claim the study was flawed due to a culture of meticulous bug-finding in Perl and Ruby?
From Conservation of Expected Evidence[1]:
If you try to weaken the counterevidence of a possible "abnormal" observation, you can only do it by weakening the support of a "normal" observation, to a precisely equal and opposite degree.
It really seems like a stretch to say that higher bug fix counts aren't due to higher defect rates, or that higher defect rates are a sign of a better language. Language communities have a ton of overlap, so it seems unlikely that language-specific cultures can diverge enough to drastically affect their propensity to find bugs.
Personally, I have read the abstract alone, and was pleased to see it nurtured my own preferences: static typing is better, strong typing is better. But then I saw this disclaimer:
> It is worth noting that these modest effects arising from language design are overwhelmingly dominated by the process factors such as project size, team size, and commit size.
So I dismissed the paper as "not conclusive", at least for the time being. I wasn't surprised if their finding were mostly noise, or a confounding factor they missed.
By the way, I recall some other paper saying that code size is the most significant factor ever to measure everything else. Which means that more concise and expressive languages, which yield smaller programs, will also reduce the time to completion, as well as the bug rate. But if their study corrects for project size, while ignoring the problems being solved, then it overlooks one of the most important effect of programming languages in a project: its size.
I'd love to see a methodology similar to the one here analyzing concurrency bugs: http://www.cs.columbia.edu/~junfeng/09fa-e6998/papers/concur... . Akin to what's done in social sciences, they applied simple labels to bugs in the bug repos -- a grad student and some undergrads can label a lot in a couple weeks -- and regress on that.
From Conservation of Expected Evidence[1]:
If you try to weaken the counterevidence of a possible "abnormal" observation, you can only do it by weakening the support of a "normal" observation, to a precisely equal and opposite degree.
It really seems like a stretch to say that higher bug fix counts aren't due to higher defect rates, or that higher defect rates are a sign of a better language. Language communities have a ton of overlap, so it seems unlikely that language-specific cultures can diverge enough to drastically affect their propensity to find bugs.
1. http://lesswrong.com/lw/ii/conservation_of_expected_evidence...