I finally found the entire Katz et al., 2010 article (http://www.ncbi.nlm.nih.gov/pubmed/20516309) and was just scanning it before running off to work. I'll look at it in depth later but I was struck by Figs. 4, 5, and 6 which only present the trend lines and not the data points.
I don't think 100 points is so much that they can't show it on one (or a few) figures... if you reduce the size of the symbol then it can easily be done. It just seems to me that only presenting the trend line will hide important facts about the variance and imply the predictive ability is far better than it is. There is a reason why we don't just report means and why we report std also as you know.
I recently plotted many hundreds of data points (in five series) on a graph (not meant for publication) to find a problem in an analytical method and also included trend lines. The trend lines were only one part of the story... the other was the HUGE variation in the parameter I was investigating that crunched down into those trend lines. In other words, you could never use that trend line to predict anything although it was clearly trending one way.
I think if you are going to only show trend lines you need to show confidence envelopes. Just based on the one bar graph in the appendix, the mean is not the message, the variation is.
And speaking of mean, do you think they should have shown median instead? Certain data I deal with are usually presented as medians and not means and I wonder if this is the case with these data. I wonder if they would have come to the same conclusions if they plotted median. In the scoliosis game, it seems the outliers are common which would drive selection of the median over the mean.
Last, it seems like the kids who were braced before the growth spurt had more progression than the kids braced after. I am not understanding why this result doesn't undermine the claim that bracing is only or most effective during the growth spurt.
I would like to see the individual data of total brace wear versus progression. I think it would show that some (many?) very compliant kids had progression and some (many?) non-compliant kids did not. It would almost have to given those error bars.
One result that will stand in all these studies is the extreme variability of response to conservative treatment even after correcting for compliance as in this case. There might be a signal buried in there by it may be impossible to find using any methodological approach.
I don't think 100 points is so much that they can't show it on one (or a few) figures... if you reduce the size of the symbol then it can easily be done. It just seems to me that only presenting the trend line will hide important facts about the variance and imply the predictive ability is far better than it is. There is a reason why we don't just report means and why we report std also as you know.
I recently plotted many hundreds of data points (in five series) on a graph (not meant for publication) to find a problem in an analytical method and also included trend lines. The trend lines were only one part of the story... the other was the HUGE variation in the parameter I was investigating that crunched down into those trend lines. In other words, you could never use that trend line to predict anything although it was clearly trending one way.
I think if you are going to only show trend lines you need to show confidence envelopes. Just based on the one bar graph in the appendix, the mean is not the message, the variation is.
And speaking of mean, do you think they should have shown median instead? Certain data I deal with are usually presented as medians and not means and I wonder if this is the case with these data. I wonder if they would have come to the same conclusions if they plotted median. In the scoliosis game, it seems the outliers are common which would drive selection of the median over the mean.
Last, it seems like the kids who were braced before the growth spurt had more progression than the kids braced after. I am not understanding why this result doesn't undermine the claim that bracing is only or most effective during the growth spurt.
I would like to see the individual data of total brace wear versus progression. I think it would show that some (many?) very compliant kids had progression and some (many?) non-compliant kids did not. It would almost have to given those error bars.
One result that will stand in all these studies is the extreme variability of response to conservative treatment even after correcting for compliance as in this case. There might be a signal buried in there by it may be impossible to find using any methodological approach.
Comment