The Braist Study

[hdugger]

This, from the paper - "The trial was stopped early owing to the efficacy of bracing." So, more or less the exact opposite of what you state here..

Here's Lori Dolan's explanation to Concerned Dad over on the Scoliosis Support forum back in 2009 about when and how they planned on stopping the study:

"Concerned Dad - We do have a set of formal stopping rules for BrAIST. After 1/3 of the required sample has reached an endpoint, we are required to present the data to the NIH and the Data Safety Monitoring Board. These data could show: 1) Bracing is so much better than observation we should stop the trial; 2) Observation is so much better than bracing we should stop the trial; 3) The outcomes are so similar between the two groups we'll never be able to find a significant difference so we should stop the trial; or 4) The outcomes are different and we should continue the trial. "

So, the endpoint turned out to be #1 - Bracing is so much better than observation we should stop the trial.

[Pooka1]

Jumping in for a few quick corrections,

Based on the discussion of Figure 2 in the report:

* 116 braced children were measured in the dose-response curve and I do not see any note of how they're broken down between randomized and unrandomized.

This point is correct. I was mis-remembering. They combined the intent-to-treat and as-treated for the dose-response graph. They had previously tabled data from just the intent-to-treat but then jumped back to all braced patients for the graph.

* The quartiles are divided by hours of wear, and not by numbers of children. You have no way of determining the number of children in each of the quartiles.

This is NOT correct as far as I can tell. Each of the 116 patients wore the brace X number of hours on average during the six months. There are still 116/4 observations. one associated with each kid, in each quartile so the top quartile is 29 kids. The top two quartiles are 58 kids. Also note the error bars.

If they don't stick with just the intent-to-treat kids, the results will be further criticized. But if they lump both groups they will lose the randomization and they already have gotten gigged on that.

On the issue of stopping the study, yes I understand why they did. I am also pointing out that they are being criticized by people in their field for not having a large enough "n." Obviously the criteria they set before hand was too low if it going to result in too few patients to have confidence in the results that draws criticism from their research community.

[Pooka1]

Here's Lori Dolan's explanation to Concerned Dad over on the Scoliosis Support forum back in 2009 about when and how they planned on stopping the study:

"Concerned Dad - We do have a set of formal stopping rules for BrAIST. After 1/3 of the required sample has reached an endpoint, we are required to present the data to the NIH and the Data Safety Monitoring Board. These data could show: 1) Bracing is so much better than observation we should stop the trial; 2) Observation is so much better than bracing we should stop the trial; 3) The outcomes are so similar between the two groups we'll never be able to find a significant difference so we should stop the trial; or 4) The outcomes are different and we should continue the trial. "

So, the endpoint turned out to be #1 - Bracing is so much better than observation we should stop the trial.

It sounds like the 1/3 criteria has come back to haunt them. Also, the hypothesis has to do with reaching 50* or not. They are not testing for surgery avoidance other than at the point of maturity. And even at that point, kids north of 45* are probably technically surgical if they are progressing even in brace.

Do you think this first BrAIST pub is a "landmark" pub?

Do you think there is no need to repeat it?

How many of the kids in the top two quartiles on the dose response curve would have to be north of 40* before you reject the connection between between being <50* at maturity and avoiding surgery for life? (I assume everyone here would outright reject all kids between 45* and 50* as likely NOT being successful in avoiding surgery... correct me if I'm wrong.)

[Pooka1]

Here's another question... how mis-matched in terms of curve types would the quartiles have to be before you decide that the results on "success" can't be taken at face value? That is, what if the majority of the T and double major curves were in the lower wear hours quartiles by chance or for some other reason? At what point would that begin to matter?

Just asking.

[hdugger]

Do you think there is no need to repeat it?

I don't believe it can be repeated in a randomized study. Ethically, they were only permitted to withhold bracing from patients in this study because there was no level 1 evidence of the efficacy. Now there is. You cannot ethically randomly assign children to *not* have an effective treatment.

. . .

There's not much point in covering the rest of this ground again. You're picking a set of conjectural endpoints out of the air and drawing conclusions/asking others to draw conclusions based on those conjectural endpoints. The standard in science is to draw conclusions based on the available, and not the conjectured, data. As more data come in, we can reassess our conclusions. Until then, it's pointless.

BTW, here's an interesting article to mull over - http://news.cnet.com/8301-1023_3-576...ments-section/

"In a post titled, "Why We're Shutting Off Our Comments," Popular Science's Online Content Director Suzanne LaBarre detailed how comments on the Internet can be bad for science at large. LaBarre cites a study led by University of Wisconsin-Madison Professor Dominique Brossard in which 1,183 Americans were given a fabricated story on nanotechnology and were asked how they felt about the subject, both before and after reading fake comments. By reading both civil and vile-trolling responses, the study found that people were swayed far more by negativity.
"Simply including an ad hominem attack in a reader comment was enough to make study participants think the downside of the reported technology was greater than they'd previously thought," wrote Brossard and coauthor Dietram A. Scheufele in an op-ed in The New York Times."

[Pooka1]

I don't believe it can be repeated in a randomized study. Ethically, they were only permitted to withhold bracing from patients in this study because there was no level 1 evidence of the efficacy. Now there is. You cannot ethically randomly assign children to *not* have an effective treatment.

Given the dismally low repeatability of those other biomedical studies, how is it ethical to NOT repeat it? How did those other studies get repeated if you are correct that they never repeat studies like BrAIST?

There's not much point in covering the rest of this ground again. You're picking a set of conjectural endpoints out of the air and drawing conclusions/asking others to draw conclusions based on those conjectural endpoints. The standard in science is to draw conclusions based on the available, and not the conjectured, data. As more data come in, we can reassess our conclusions. Until then, it's pointless.

All good science is hypothesis driven. You are arguing against that and waiting until the results come in and then assessing it. That is one of the reasons why most published biomedical research is false.

Eventually we will learn the final curve measurements. I am asking people if they can decide on a threshold BEFORE that happens so they can honestly assess the results. If you do it after then you are probably being swayed by the results.

[Pooka1]

BTW, here's an interesting article to mull over - http://news.cnet.com/8301-1023_3-576...ments-section/

"In a post titled, "Why We're Shutting Off Our Comments," Popular Science's Online Content Director Suzanne LaBarre detailed how comments on the Internet can be bad for science at large. LaBarre cites a study led by University of Wisconsin-Madison Professor Dominique Brossard in which 1,183 Americans were given a fabricated story on nanotechnology and were asked how they felt about the subject, both before and after reading fake comments. By reading both civil and vile-trolling responses, the study found that people were swayed far more by negativity.
"Simply including an ad hominem attack in a reader comment was enough to make study participants think the downside of the reported technology was greater than they'd previously thought," wrote Brossard and coauthor Dietram A. Scheufele in an op-ed in The New York Times."

There is a MUCH better reason to not have comment sections on technical matters... most/all people commenting have no relevant training or understanding to be commenting.

I have been banging on and on for years about how ridiculous it is for this forum to have a research section. There is exactly one person who has been on here who is known to have relevant training in this field. The only legitimate use of the research section would be to have that researcher posting stuff. One person. At that point, why bother?

What I have been doing is in the few general science areas, I can try to comment. That is MUCH easier than doing the research in this area. Anyone with science training can make the comments I make. In general, review is much easier than the actual research. But beyond those few general science issues that can be picked out, I quote actual authorities in the field. Of course those are conversation stoppers as we have seen until someone posts an opposing quote from another researcher which is extremely helpful to let people know the range of thought among researchers in a given field. That is valuable.

[Pooka1]

$1.3 million to validate 50 major cancer biology studies

And that is cancer which is much more likely to be life to death than scoliosis.

http://news.sciencemag.org/funding/2...cancer-studies

As several scientists told Science last year when the initiative got off the ground, Iorns is tackling a significant problem in biomedical research: the fact that many published studies can’t be repeated, and that many researchers aren’t enthusiastic about simply replicating what someone else has already done. As she stated earlier this week, the new funding will be critical in “helping to institutionalize scientific replication.”

Not repeating studies like BrAIST is a part of why most published research results are false. See articles mentioned upthread for why.

If BrAIST went the other way and showed no real difference between the observation and bracing, it STILL would need to be repeated. You have to grasp how hard it is to do this research.

[hdugger]

I am asking people if they can decide on a threshold BEFORE that happens so they can honestly assess the results.

There are only two populations who need to decide on a threshold, and you and I are not in either.

Doctor's need to decide whether it makes sense to prescribe a brace, in a specific case, and families being offered braces need to decide whether or not to accept that treatment.

Everyone outside these two populations is just standing on the sidelines kibitzing. Which would be no nevermind except that, according to the research, their negative comments are unduly influencing at least one of the populations trying to make the decision.

[hdugger]

This is NOT correct as far as I can tell. Each of the 116 patients wore the brace X number of hours on average during the six months. . . The top two quartiles are 58 kids.

I'd decided the other way because of my preconception that patients generally did not manage to wear their braces for that many hours a day. Apparently that's wrong, if half of the kids managed to wear them for over 13 hours a day for 6 months, and likewise managed to avoid surgery in 90% of the cases. So, only 1/4 wore them infrequently? That sort of surprises me, but I'm willing to accept it as so.

If BrAIST went the other way and showed no real difference between the observation and bracing, it STILL would need to be repeated.

Yes, they could repeat it if no real difference was found. Then it would not be considered unethical to withhold the treatment, since the treatment was not proven to be effective. It's only when a treatment is found to be effective that they can't simply randomize patients to not receive it.

[Pooka1]

There are only two populations who need to decide on a threshold, and you and I are not in either.

Of course. We are lay people. This is obviously just for learning purposes.

Everyone outside these two populations is just standing on the sidelines kibitzing. Which would be no nevermind except that, according to the research, their negative comments are unduly influencing at least one of the populations trying to make the decision.

That's why I quoted several researchers in this field upthread. Their cautionary comments should probably be considered to be taken on board, yes?

[Pooka1]

Wait a minute... are you including my pointing out that the final curve measurements are not known is a "negative" comment?

[Pooka1]

Yes, they could repeat it if no real difference was found. Then it would not be considered unethical to withhold the treatment, since the treatment was not proven to be effective. It's only when a treatment is found to be effective that they can't simply randomize patients to not receive it.

But it is known that only a small percentage of these studies are able to get the same result when you repeat it. The conclusion can still be wrong. We are talking hard bracing. Why does Dr. Hey say bracing, UNlike chemotherapy for cancer, or appendix removal for appendicitis, is STILL a choice?

Another surgeon I quoted openly wondered if these results would change the treatment. Apparently at least some experts in this field are less persuaded by the study than lay people.

[Pooka1]

I'd decided the other way because of my preconception that patients generally did not manage to wear their braces for that many hours a day. Apparently that's wrong, if half of the kids managed to wear them for over 13 hours a day for 6 months, and likewise managed to avoid surgery in 90% of the cases. So, only 1/4 wore them infrequently? That sort of surprises me, but I'm willing to accept it as so.

By the way, these types of graphs may be commonplace in the medical literature (I don't know) but I wonder if these results can be presented in a more straightforward fashion where lay people don't have to guess what is being shown.

[hdugger]

""Simply including an ad hominem attack in a reader comment was enough to make study participants think the downside of the reported technology was greater than they'd previously thought," "

plus

There is something funny about them stopping the study early . . . That is one possible explanation why they didn't publish the bottom line data. It smells of politics. Stopping the study early at the point it was stopped for the reasons stated is objectively bizarre as far as I can tell. . . . There is something funny here. The authors are going to be taken to task on this. . . . I am hoping a political stink has not invaded this study . . . Imagine the fall out . . . Would there be any credibility left whatsoever?

= readers "thinking the downside of the reported . . . was greater than they'd previously thought"