Appendix: Bad Science Bingo in the BMA’s “safe cycling” pages

This is just a crude brain dump of a post that comes after the serious series — posts one, two, three, four, five, six, seven and eight.

Sorry, I just can’t get over these extraordinary pages on the BMA’s website. Here’s a very quick run through some of the Bad Science Bingo points that leaped out.

There were the canards, fallacies, and methods of misdirection:

  1. Obviously there’s the emphasis on anecdotes and cases, the lowest form of evidence, which are essentially appeals to emotion.
  2. Coupled with that the description of the “beliefs” of a few doctors, designed to nudge readers into conformity (acting as a subtle argument from authority for readers who are not doctors, and an argumentum ad populum for those who are).
  3. Specifically in a couple of the anecdotes the selective recall of serious injuries in non-helmet wearers and minor injuries in helmet wearers (creating the illusion of control).
  4. “Figures from New Zealand show that in 2006 there were 883 cyclists injured and nine killed. This corresponds to 20 people per 100,000 injured and 0.2 people per 100,000 killed. These figures are lower than those reported for 1994 when legislation was first introduced.” Fun factoids, but they don’t actually say anything about helmet efficacy. Lots of things changed between 1994 and 2006. (Post hoc, etc.) Perhaps there is evidence for NZ’s legislation improving safety but the 2006 crude injury statistics aren’t it.
  5. Incidentally, while we’re on correlation and causation, the authors even get their statements on cycle tracks subtly wrong: “During the period of 1976 to 1995 Germany almost tripled their mass of cycle networks and this led to a 64 per cent drop in cyclist deaths.” While the evidence of a causative link is much stronger here, it’s a lot more complicated that a simple one “led to” the other. The reference does indeed state that Germany tripled their cycle network and that their death rate fell, but it notes that the later is in part the result — directly and indirectly — of the former.
  6. I loved this statement, when discussing the side-effect of reduced rates of cycling: “If legislation were to reduce the rates of serious injury and promote increased public confidence in cycling, the effect might be to make cycling more popular. Clearly, there is a need for further research on this matter.” I don’t know where to begin. After dismissing all the side-effects of helmets as being based on too weak and preliminary evidence, the BMA counter it all with a speculation based on none at all — and tell us that there is a clear need for more research. Well quite.

And there were specific claims or activities that run counter to the cited evidence, or subtly misrepresented it (I did not systematically check references, these are simply things that leaped out as contradicting what I recall of the literature):

  1. On page 2 the BMA list the things they are doing in addition to promoting helmets. The first item is “publicity and education campaigns in order to raise drivers’ awareness of more vulnerable road-users, including cyclists”. We know that these don’t really work.
  2. The “risk compensation” section on the fifth page cites just one source, the Spanish study described on Monday, whose study design we know can not answer the question that they are asking it to answer.
  3. “As noted in Table 2 the Macpherson and Spinks 2007 Cochrane review found no evidence to either support or counter the possibility that legislation may lead to negative societal and health impacts such as reductions in cycling participation.” You would probably read this and think, “studies have been done and they found no evidence for X.” It actually means, “the studies didn’t bother looking at X.”

And there were fun inconsistencies:

  1. Kirsty’s story on page 5, “Doctors believe that had she not been wearing a cycle helmet at the time of her crash, she would have died,” and on page 6, “They have been shown to reduce the risk of head injury and its severity should it occur. This does not apply to fatal crashes but in such instances the force of impact is considered to be so significant that most protection would fail.”

The resource is just generally bizarre. It has a very weird set of focusses. On one page it gives a seemingly arbitrary selection of factoids from cyclist demographics (notably absent is any acknowledgement that “cycling” is not a single activity); on another it notes the diversity of cycle helmet standards — but fails to discuss any of the important consequences of this, such as how few helmets these days meet the stricter standards that applied in the past, back when most of the evidence on helmet efficacy was collected. In a table on the fifth page they mention that a study found no evidence of helmets causing or exacerbating rotational injuries — yet this is the only mention they make of the rotational injuries problem. Their inclusion and omission criteria appears to be completely random.

Anyway, enough of this. I don’t want to hog the game — your turn.

About these ads

4 responses to “Appendix: Bad Science Bingo in the BMA’s “safe cycling” pages

  1. Mike Chalkley - Chair Bournemouth Cycling Forum

    Excellent series of articles – I was doubly shocked at the bill Annette Brookes is proposing when I saw it was backed by the BMA. The Bournemouth Cycling Forum has a rep from the Primary Care Trust, Lindley Owen, who I emailed for a comment. He pointed me to some work by Dr Adrian Davis which can be found at http://www.bristol.gov.uk/ccm/content/Transport-Streets/Walking-Cycling/cycling-in-bristol/essential-evidence/essential-evidence.en
    Read number 25 – it’s a good critique of evidence.
    Lindley wrote…
    “From the point of view of the public and their representatives, however, the evidence is not enough. Consider; if you allow your child to walk and cycle regularly, they are likely to grow up as fit, confident, socially integrated adults. In your dotage you will be congratulated for your success. On the other hand, if your small, precious, irreplaceable child is killed or (almost worse) brain damaged because you didn’t wrap them in polystyrene, you will become an instant pariah in your community.
    Telling MPs the evidence is never going to be enough. When they’re on a crusade, they have vested a lot of personal credibility in an issue, and have attracted media attention which is essential to their continued existence. What we can do is use the heightened awareness to show where the real problem lies.”

  2. All the intellectual rigour of a homeopathy leaflet.

  3. My personal fav was:
    “No evidence was found to either support or counter the possibility that legislation may lead to negative societal and health impacts such as reductions in cycling participation. ”
    Except for the study listed 4 paragraphs earlier – Robinson DL
    (2006) where “before and after data show enforced helmet laws discourage cycling but produce no obvious response in percentage of head injuries”

  4. Pingback: All those helmets posts in one place | At War With The Motorist

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s