This went down like the proverbial lead balloon. I had overlooked the fact that several members of the medical department and even some of biostatistics count their publications in JACASS as being amongst the jewels in their scientific record. I can’t understand it myself since I always make damn sure that my name never gets stuck on any paper heading for a journal with innumerate readers. By the time the moronic editors and referees have hacked it to bits there is nothing of any value left anyway. Anyway, I don’t think I have been so unpopular since I refused to validate the spreadsheet of one of our biochemistry units on the grounds that the program clearly didn’t work. “How can you tell?”, they asked, “since you haven’t even looked at it”. “I don’t need to look at it. It’s obviously wrong and I am just applying some Bayesian inference, and a very informative prior”, I replied. “It’s the combination of numerically challenged scientists, moderately complex application and spreadsheets that has probability zero of producing a valid result”. They were outraged and the powers that be forced me to look at the damn thing. Luckily, I found the negative sum of squares in the first 30 seconds.
Anyway, to get back to JACASS, my punishment from on high, for giving too much lip, was to force me into academic interaction. I was sent to see Professor Percy Vere, a physician in the Department of Evidence Based Medicine at the University of Notchester who is known throughout the Archie Association for his work on meta-analysis. We had done a meta-analysis ourselves of Detense®, a treatment for hypertension of ours (rather more effective than Tensoff®) and Vere had done the same. The two analyses did not agree and I was being seconded to try and sort things out.
I can’t claim that our first meeting was much of a success. I don’t know who had briefed Vere but he seemed to think that I has been sent to learn how to conduct a meta-analysis according to the procedures of the Archie Association, about which I couldn’t give a damn, whereas the real reason I was there, surely, was to bring the light of statistical reason into the heathen darkness of medical research.
“You see, Mr McPearson”, he said, “the important thing is that any meta-analysis should be produced in such a way that it would satisfy the MVT.”
“The MVT, Mr Vere?” I replied.
“That’s Professor Vere, Mr McPearson” he said, “and MVT stands for Meta-analysis Validation Tool.”
“That’s Dr McPearson, Professor Vere,” I said, “and I presume that you are referring to the index proposed by Gaucho and Roper in JOG.” (JOG is the Journal of Overviewing and Generalization)
“There’s no need to stand on your dignity,” he said, “and yes, it is the Gaucho and Roper1 index and if you are acquainted with it I am sure that you will see its value.”
“Very valuable as far, as it goes,” I replied, “but unfortunately not far enough.”
“How do you mean?!” said Vere in outraged tones.
“Well, I don’t see any of the following. Did the analysis respect the structure of the original trials? Was double counting avoided? Was data-imputation necessary? Was pooling carried out in a way that was relevant to the question put? ”
“I am afraid you will have to explain the relevance of these points, Mr McPearson.”
“Right. Here goes. You see this meta-analysis?,” I said, picking up a paper from my briefcase that had been published recently in the Albion Physcian’s Enquirer (APE) and holding it at arm’s length suspended from one corner before placing it on the table, carefully, and opening it. “You will note that it synthesises a mixture of cross-over and parallel group trials. Yet you will see, however that the data are summarised in terms of means in each group and standard deviations. Yet this information could not allow you to analyse a single cross-over trial adequately, so it beats me as to how it could be used to summarise results from cross-over trials adequately.”
“What a silly objection. This is just the way that the software package we use, MetOrg, requires the data.”
“So much the worse for it. Now have a look at the standard deviations from those cross-over trials that I have highlighted. You will note that they are all equal to 10.0 for each arm and that there appear to be 16 such values. How do you explain this extraordinary coincidence?”
“Oh that’s very easy. We are always having to do this sort of thing. You see, very often people who publish cross-over trials fail to give the appropriate information. They will insist on producing estimates of the treatment difference and the associated standard error or confidence intervals, rather than the treatment means and standard deviations, so that we have to use plausible values instead. But after all, all things are related, we can use values estimated from the other trials. No man is an island etc,” and then adding with barely concealed mirth, “although some appear to be a Channel Island.”
“Very revealing,” I replied with barely concealed distaste, “I suppose that you could call this an example of Vere-say evidence.”
“I do think that making puns on people’s names is very cheap,” he replied with glacial hauteur. “What were your other points?”
“Well you will note”, I said, “that even for the parallel group trials the same means and standard deviations appeared twice in the control group arm in many places.”
“That’s an obvious point. We often find this. These must be three-arm trials so the control group has been entered twice, once for each experimental arm. This is very common.”
“Yes. So it appears and this brings me to my final point. Why is it necessary to pool the results from two experimental arms from one trial into a meta-analysis? After all if there are different arms, the treatments are different.”
“This is also very common. Why, we did it in our meta-analysis of Detense. I think you will find that it is merely that results for different doses have been pooled.”
“Call me old-fashioned,” I said, “but as someone who actually works in drug development, I consider that differences between doses are important. A lot of what we do is geared to finding appropriate doses.”
“Well this is a very minor consideration. The important thing is to find all the evidence. Nothing you have shown me from this paper seems inappropriate. Let me have a look. This is outrageous!!!!. This is our meta-analysis of Detense. Surely you are not going to argue with this? It’s pear reviewed, carried out according to Archie Association guidelines, designed by distinguished academic experts and published in a prestigious journal. It’s beyond reproach. How would you go about doing such a meta-analysis?”
“Well, we have gone about it,” I said, “which is why I am here. We have our own meta-analysis in which we have not pooled different doses, have not counted control arms twice, have used standard errors and treatment contrasts from given trials, thus enabling us to include cross-over trials properly and have not imputed any values, since we have access to all relevant data.”
“I would be very interested to know how you have managed to do this using MetOrg,” said DeVere.
“We haven’t,” I said, “we have programmed it ourselves”.
“Programmed it yourselves! That can’t be a valid approach. What hubris. What makes you competent to carry out this sort of scientific innovation? Who will accept that? Are you aware that published analyses in APE (the Albion Physician's Enquirer) show that the standard of pharmaceutical industry meta-analyses is inferior to Archie Association ones.
“Would this be research carried out by members of the Archie Association using the Gaucho and Roper quality index?”
“Yes. So you need have no fear about the validity of the result.”
Well that was it. My best attempts at diplomacy and persuasion had failed. It was impossible to get the man to see reason. In the end, he proposed that we get an independent assessment by a third party to see who was right and I, unwisely, agreed.
Some weeks later I passed Dr Angina Cutter in the corridor.
“Guernsey,” she said, “good news regarding the Detense meta-analysis”
“Oh,” I replied, “Vere and I agreed that we would get a third party to review our and his analyses. He was going to get back to me with a proposal for a name.”
“He’s done better than that,” trilled Angina delightfully, her eyes shining , “He’s gone ahead and had the review done. I was talking to Sir Lancelot just now (what a charming man) and he was telling me that Pannostrum and the Archie Association are now singing from the same hymn sheet as he put it and that we have agreed the results. Isn’t that great? Another problem solved.”
“I presume,” I said gloomily, “that that means we are accepting Vere’s results.”
“Yes, apparently. Not our fault. It seems we were using the wrong software. It seems that we should have been using MetOrg. Had we been doing so, we would have had to do it properly. I proposed to Sir Lancelot that we should make it company policy to always use MetOrg in future and the good news for you is that he agreed that this was a good idea and we will be obtaining copies of MetOrg.”
“And this independent statistician,” I said gloomily, “is not perchance someone who works at the University of Notchester?”
“Yes. How did you guess?”
“Something to do with a fundamental statistical concept Vere was having difficulty with,” I said.
“Oh. What was that?”
“Independence.”