Search This Blog

Showing posts with label physiotherapy. Show all posts
Showing posts with label physiotherapy. Show all posts

Thursday, 22 November 2018

If Dr Who can't help youngsters who have dyspraxia, who will...

Dr Who?

Dr Who. She is a Time Lord with a friend who has dyspraxia. This isn't about Dr Who. It's about children with poor balance and coordination, who struggle with basic movement skills that we all take for granted.



Dyspraxia/Developmental Coordination Disorder


A good number of children in the UK have dyspraxia: difficulty with movement, balance, and coordination generally. Basically, they have very poor fundamental movement skills.


In the absence of any other diagnosis that would affect motor ability, children with poor movement skills may be diagnosed with Developmental Coordination Disorder (DCD). This affects up to six children in every classroom, and is a diagnosis usually given quite late: children's motor skills are still developing aged six or seven e.g. at the more difficult end of  human motor activity such as balancing on a narrow beam and other agility or gymnastic activities. For this reason, clumsiness might simply be down to late developers and thus diagnosis is delayed. One physiotherapist I know, whose child was diagnosed with DCD, believes that children with DCD are simply well below average. After all, we can't all be Nadia Comeneci.


Poor movement skills have a profound and devastating impact on children's development and life chances. There is a large body of evidence showing how poor movement skills affect children's physical, emotional, social, psychological  and educational development. Researchers have called children with movement difficulties 'onlookers in the playground', and one study found that a large number (80%) of seven year old children with DCD went on to experience drug and alcohol abuse, trouble with the law and mental health difficulties by the age of 22, compared to 11% of the children with typical movement skills. Further evidence also suggests that children’s motor skills have a positive association with physical fitness, body weight and physical activity. Because physical activity tracks across the lifespan, increasing physical activity in childhood can have huge benefits in population health (including mental health) and academic achievement.



Reducing physical inactivity could therefore reduce morbidity but it could also reduce premature deaths and save money. In 2006, research showed that physical inactivity caused 16.9% of premature deaths and costs £8.2 billion per year. However, only 22% of children meet Department of Health recommendations for physical activity, figures which are worsening over the years.  One in five children in Reception and one in three children in Year 6 are obese or overweight. In 2007 obesity cost the economy  £15.8 billion per year, including £4.2 billion in direct costs to the NHS.

In Bradford, a large cohort study (Born in Bradford) has been following over 13,500 children from birth since 2007. During investigations and assessments in schools, we have found that a large proportion of children experience difficulty with their fundamental movement skills. These difficulties have been noted by children's teachers, who have repeatedly asked if there are any exercises that would help these children. For this reason we conducted a systematic review looking for evidence-based activities which could benefit children with poor movement skills. This found nine studies which investigated a total of 16 interventions. Three well-conducted trials in particular found that some physiotherapy activities produced large effect sizes in outcome measures evaluating the children's movement skills.


I have developed these evidence-based activities into a physiotherapy programme designed to be delivered in schools by school staff.





This overcomes the problem for these children of having to wait until formally diagnosed with having a problem before they can be referred for help with physiotherapy or occupational therapy. Besides, there is a sting in the tail for these neglected children because they do not necessarily receive the treatment they so badly need even then. Even before austerity kicked in and reduced numbers of NHS therapists, children with poor movement skills made up 60% of occupational therapy team's waiting lists, and some had to wait up to four years just for an assessment! This is unlikely to have improved in the years of austerity experienced by the NHS since 2007.


Teachers recognise that there is a relationship between poor movement skills, balance and coordination and academic achievement. My qualitative research at a PE teachers and head teachers conference, at meetings with head teachers and from piloting of the evidence-based activities in a school suggests that head teachers and school staff are keen and willing to deliver a programme like this. Indeed, there are already programmes in several schools across the country, but these are not evidence-based or properly evaluated for efficacy.



What are we waiting for!? Let's get it tested !


Unfortunately, despite a number of pilots in schools and some work with teachers and non-governmental organisations, I have been unable to secure funding for an appropriate study to evaluate whether the programme can successfully be delivered in schools by school staff. In the long term, a multi-centre randomised controlled trial (RCT) to evaluate the effectiveness of the physiotherapy programme is necessary, but I first need to conduct a feasibility study to find out whether the programme can be delivered at all by school staff. This feasibility study will also gather essential data needed for properly designing the RCT. 




This research has follwed the Medical Research Council guidelines for developing complex interventions, a process that includes developing the scientific rationale, conducting a systematic review, piloting and then conducting a full trial if justified. So, according to the NIHR definition, a feasibility study finds out 'can this be done?', and does not necessarily need a randomised element. However, I will be evaluating change in children's movement skills, because if there is no change, then a multi-centre randomised trial will be unnecessary. My colleagues are developing a new measure of fundamental motor skills which I hope will be fully tested and published when this research is eventually conducted.


So, what's the problem?


My funding applications have been rejected at the final hurdle, after passing through earlier stages satisfactorily. This has proved extremely frustrating, especially given that I was confident because of the outstanding advice, support and guidance from my awesome mentors, the NIHR's Research Design Service and some senior professors.


The support and positive feedback of senior and highly respected academics gives me confidence in my research, although it does take a battering. One has to be thick-skinned in academia; with reviewers of applications and papers submitted to journals for peer review protected by anonymity, comments and feedback can be pretty brutal, and often just unprofessional or even occasionally stupid. One reviewer (of an NIHR application) wanted to know what  a feasibility study was, even though I had referenced the NIHR's definition and description with every mention of 'feasibility study'.




Well, it is a feasibility study. So, it's at the feasibility study stage.




At one interview, I had to explain the purpose of a control group to a professor.  Another reviewer was of the opinion that it shouldn't be funded in case the children didn't get faster and their hopes were dashed. I hope that reviewer doesn't review any proposals for research into any promising cancer treatments.

In case I look as though I feel hard done by, check out Professor Carol Greider. After receiving news earlier in the day that she was being awarded the Nobel Prize for Medicine, Professor Greider was rejected for a grant by a panel who questioned whether she really knew what she was doing -  in that very area of research for which she was awarded the Nobel Prize! My own professor was once told to get a native English speaker to read through his application before submission.



So the process of applying for funding is frustrating, daunting, painful, time-consuming and occasionally surreal. It takes a thick skin, tenacity and a sense of humour. It takes months to complete some applications. Research is hard, getting funding even harder - but life is hard for children with dyspraxia, and I am determined to see the movement skills programme further developed and tested. It could contribute to better lives for disadvantaged children across the UK and beyond, as well as have significant benefits for population health and the economy.



Saturday, 6 October 2018

Putting the CAT (Critically Appraised Topic) among the pigeons

The Epiphany Club

Research and clinical physiotherapists in Yorkshire have started meeting on a regular basis for mutual support, to share ideas and to improve their capacity for communication and working together.  This is the Epiphany Club, an informal group of physios and other AHPs gathering every few months in a pub near to Leeds Town Centre. 



Click to enlarge


Critically Appraised Topics

One well-received proposal for the Epiphany Club was the development of Critically Appraised Topics (CAT). Keele University School of Health and Rehabilitation has produced a number of very useful musculoskeletal CATs and, over the last fifteen years, their collaboration with clinicians has developed into a hugely beneficial, productive and closely-knit team. A CAT is: 
"a summary of the best available evidence, which answers a clinical question and includes a clinical ‘bottom-line’. A CAT is essentially patient-based, in that it begins with a clinical question generated from a specific patient situation or problem."

This blog will describe the development of the first CAT and its clinical bottom line, generated from the best available evidence. 

Background

No sooner had the CAT idea been put forward than a clinical problem cropped up that was ideal for a CAT. The problem had arisen in a ward-based neuro rehabilitation setting and was causing some anxiety among members of the neuro rehab team. 

After mobilising a patient with an acute partial spinal cord injury, a registrar who had worked at a specialist spinal centre suggested that mobilising patients was inappropriate. He described the current protocol at specialist centres as six weeks' 'bed rest' (or 'Active Bed Based Rehabilitation'). In contrast, the standard procedure on the Leeds Trust rehabilitation unit was early mobilisation when safe to do so (i.e. when medically and surgically stable). But what evidence was there for or against either approach? This then was a clinical question generated by a problem with a specific patient situation, and ideal for a CAT.


Further information: Sir Ludwig Guttmann

There is no doubt that Sir Ludwig Guttmann was a remarkable physican and inspirational human who transformed the quality of care, quality of life and extended the lives of people with spinal cord injuries. The prevailing thought pre-1945 was that rehabilitation was "irrational" and death was inevitable within months, but the principles that he laid down in 1945 were responsible for transforming the outlook and prognosis of people with spinal cord injury. These principles included, for example, the routine turning of patients to prevent skin breakdown. They have remained apparently unchallenged as core practice until at least 1979, but while nobody suggests that turning bedbound patients is not best practice, what is the case for bed rest or against mobilising?

Clinically Appraised Topic (CAT): mobilising acute partial Spinal Cord Injury patients

This Clinically Appraised Topic (Foster et al., 2001) was conducted by Leeds Teaching Hospitals NHS Trust Neurological Rehabilitation team, the Academic Department of Rehabilitation Medicine (University of Leeds) and Leeds Teaching Hospitals Library & Evidence Research Centre, It is described using the the format of the Keele University CATs. 

Acknowledgements

The CAT team would like to thank Kay Stevenson (Consultant Physiotherapist and Clinical Champion, Honorary Lecturer, School of Health and Rehabilitation, Keele University) for her guidance and support leading up to and during the development of this Clinically Appraised Topic.

Specific question 
(click to enlarge)


Clinical bottom line

There was no evidence against mobilising people with partial spinal cord injury (SCI) who are medically and surgically stable less than six weeks after injury. There were no adverse events reported in stable patients with SCI who received early mobilisation.

There was no evidence that bed-based rehabilitation improves outcomes in comparison to early mobilisation in stable patients with partial SCI.

Why is this important?


Once medically and surgically stable, people with acute partial SCI are managed differently in the first six weeks of their rehabilitation depending on the opinion of the professionals caring for them as to best practice. Some centres favour early mobilisation and others support a period (at least six weeks) of active bed-based rehabilitation.

Inclusion criteria

Patients with acute partial Spinal Cord Injury.

Search strategy 



Search terms and databases
(click to enlarge)


Papers found and selection filter


Flow chart 
(click to enlarge)


Summary and assessment of quality of papers

Quality of papers assessed using CASP Systematic review checklist(Critical Appraisal Skills Programme, 2018b) and RCT checklist(Critical Appraisal Skills Programme, 2018a)


Click to enlarge

Click to enlarge


Click to enlarge 
Click to enlarge

Click to enlarge


Summary

There is no evidence that early mobilisation is harmful for acute SCI patients who are medically and surgically stable. Studies are of generally poor quality and do not directly answer the CAT question. 

Evidence from one high quality RCT suggests that overground mobility training is better than body-weight supported treadmill training in acute (2 – 26 weeks) partial SCI patients. There was a positive effect of early exercise on muscle tissue within the first 3–6 months post SCI but these were poor quality studies.

There were weak recommendations in one poor quality paper for early rehabilitation in patients with traumatic SCI when they are medically stable and can tolerate treatment.

There is no evidence supporting six weeks of bed rest.


Suggestions for research

This CAT suggests that properly-conducted research is needed to investigate which approach has better outcomes for people with acute partial spinal cord injury. Given numbers of injuries, this would need to be a large multicentre trial exploring multiple outcomes and long term follow ups. 

References

BAGNALL, A. M., JONES, L., DUFFY, S. & RIEMSMA, R. P. 2008. Spinal fixation surgery foracute traumatic spinal cord injury. Cochrane Database Syst Rev, Cd004725.

CRITICAL APPRAISAL SKILLS PROGRAMME. 2018a. CASP Randomised Control Trial Checklist [Online]. Available: https://casp-uk.net/wp-content/uploads/2018/01/CASP-Systematic-Review-Checklist.pdf [Accessed 01 October 2018].

CRITICAL APPRAISAL SKILLS PROGRAMME. 2018b. CASP Systematic Review Checklist [Online]. Available: https://casp-uk.net/wp-content/uploads/2018/01/CASP-Systematic-Review-Checklist.pdf [Accessed 01 October 2018].


FOSTER, N., BARLAS, P., CHESTERTON, L. & WONG, J. 2001. Critically Appraised Topics (CATs). Physiotherapy, 87, 179-190.

MEHRHOLZ, J., KUGLER, J. & POHL, M. 2012. Locomotor training forwalking after spinal cord injury. Cochrane Database Syst Rev, 11, Cd006676.

PANISSET, M. G., GALEA, M. P. & EL-ANSARY, D. 2016. Does earlyexercise attenuate muscle atrophy or bone loss after spinal cord injury? Spinal Cord, 54, 84-92.

RICE, L. A., SMITH, I., KELLEHER, A. R., GREENWALD, K., HOELMER, C. & BONINGER, M. L. 2013. Impact of the clinical practice guideline forpreservation of upper limb function on transfer skills of persons with acutespinal cord injury. Arch Phys Med Rehabil, 94, 1230-46.



Thursday, 19 January 2017

What should allied health professionals do when their patients ask them about 'alternative' medicine?

Introduction

'Alternative' medicine is a huge business. Boiron, who operate globally selling homeopathic products, made a profit in 2015 of €73.9 million. Cupping as an 'alternative' therapy was given a huge boost in publicity during the 2016 Olympics, when athletes were noted to be covered with the large characteristic bruises.

So what? People can spend their money on anything they like. T
he Daily Mail, Croc shoes, jeans worn so low they reveal that the wearer's gluteal fold, detox diets and other fad diet regimes, Nestle products... facial tattoos?







Health care professionals are expected to be non-judgmental, after all. But we are health care professionals with a duty of care, firstly to our patients and secondly to the health of the nation, for whom we need to apply "the evidence of what we know works to achieve the step-change in the nation’s health that we all seek."


Is it, therefore, the duty of health care professionals to inform and advise members of the public about the evidence for, and likelihood of, benefits and risks of Complementary and 'Alternative' Medicine (CAM), particularly our patients who ask us or tell us about their use of  CAM?

We have a Bachelor of Science degree; we have expertise in functional anatomy, physiology, and mechanisms of injury, disease and repair. Surely we are qualified to comment, and obliged to do so?


I believe that CAM with no evidence to support its efficacy should not be used in either the NHS, or within physiotherapy practice - NHS or private. But there is evidence suggesting that CAM is widely used within the profession and that our Bachelor of Science degree does not prepare us adequately for appraising the evidence. This is a problem for the health of our patients and the profession itself.

Why should we say anything?

Before I became immersed in research, I worked as a musculoskeletal physiotherapist and as a rotational children's physiotherapist. In both roles, my patients (or parents of children) often asked for advice about the relative merits of 'alternative' medicine but I felt that I wasn't well equipped to answer these questions.

But then one family were keen to try an alternative therapy that promised much greater benefits than those proposed by medical treatment and physiotherapy. The family, desperate for help for their child with a rare and profoundly disabling condition, was low income but willing to do anything - including find and spend the several thousand pounds that this treatment would cost. 

On investigation, I found that the treatment was based on a premise which contradicted the currently established mechanisms of neurological repair; it suggests that improving breathing in disabled children would permit "restorative brain growth" in damaged neural tissue. At extra cost, time spent in an oxygen tank containing 100% oxygen would increase this growth further. There was no research investigating this technique other than an unpublished exploratory study which found no functional benefits (this remains the only research on this technique today). I advised the family that there were unlikely to be benefits from the treatment, based on its proposed mechanism, and that there was no acceptable levels of evidence for any benefits. I also found an anecdote on the Chartered Society of Physiotherapy's interactive forum from a physiotherapist whose caseload included a family already undertaking the same technique. This family, as part of their 'therapy', were told to withdraw from all physiotherapy and associated interventions such as standing frames - the child's physical condition deteriorated to such a severe extent that they became the subject of a Child Protection case.

This anecdote and my experience with the family alarmed me, and I subsequently explored different 'alternative' medicines. I dislike 'alternative' medicine as a description. It isn't an alternative to medicine unless it has been scientifically tested and shown to work - in which case it becomes medicine. Dara O' Briain puts it far better than I ever could, and if you have time, Tim Minchin's song Storm also is worth ten minutes. 





But that's not all. It's scarier than that


Why should health professionals get involved? What's the harm? 

Firstly, a former colleague explained to me that she has no problem with her patients seeking alternative medicine interventions because "they do no harm." 

This assertion is arguable, if not demonstrably misinformed, and I believe that as health care professionals we have a duty of care to inform the public of potential harm, as well as more effective options. For more examples, homeopaths sell prophylactic malaria kits - pure quackery - and chiropractors inform parents who have their babies' spines 'adjusted' that the babies do not then need their vaccinations.

The Chartered Society of Physiotherapy supports quackery

Some physiotherapists are heavily involved in what is as far from evidence-based practice as one can get. There is even a Chartered Society of Physiotherapy-funded Clinical Interest Group for Energy Medicine. 

Energy Medicine was shown to be charlatanism by a remarkable young lady who is the youngest person ever to publish in a peer-reviewed medical journal. Aged just nine years old, Emily Rosa designed and carried out a school science project, an elegant single-blind study on 21 energy medicine (therapeutic touch) 'healers'. The results showed that energy medicine  'healers' can detect an energy field no more than by chance. 

Can you imagine, if you'd never heard of reiki or energy medicine, and you were supervising a student on a musculoskeletal or neurology or respiratory placement, and you asked what they were planning to do with the low back pain/acute stroke/consolidated lung patient. "Oh, I am just going to hold my hands near them and direct healing energy at them."

Energy medicine: literally, 'wishful thinking'. 





It won't surprise many physiotherapists that there is also a Clinical Interest group for acupuncture, a modality which has been subjected to thousands - I think over five thousand - clinical trials. There is still no evidence that it actually works and NICE guidelines now state that acupuncture should not be offered as a treatment option for low back pain. 





NHS physiotherapy teams providing quackery on tax payers' money

According to a thread on the interactive CSP website, an entire team within an NHS Trust is about to start using cupping. The thread suggests that cupping is already in widespread use with physiotherapists.

Cupping is chiropractic nonsense, and this take down and ridicule of cupping AND chiropractic is satisfying for those of us who despair of the gullibility of the press and, sadly, health care professionals who adopt and use these unproven, fanciful therapies. ⚠ Warning: it is probably painful for those of kind and empathic disposition. Even I felt a twinge of compassion for the chiropractor until I remembered he calls himself a doctor, fleeces people, and advises parents not to vaccinate their children. 

They also perform grade 5 manipulations of babies' spines, (at 3 mins 18 seconds - again, not for the faint of heart, it makes me feel physically sick). 




Lack of critical thinking? Lack of applied knowledge? Gullibility?

As with chiropractic, homeopathy, and most forms of 'alternative' medicine, the rationale for craniosacral therapy is nonsense - the light touch (of eight grams) of a practitioner's finger tips manipulating the (fused!) skull bones of the recipient to affect a cure of all sorts of medical problems. A comparison of three physical therapists who used craniosacral therapy to evaluate 'craniosacral rate' (their diagnostic variable) of 12 children all found different rates for each child (their inter rater correlation was -0.02). In other words, each therapist diagnosed a different condition for each child.

In reality, even if I were to bang my head repeatedly against the desk (something I feel like doing when I read about physiotherapists embracing some of these alternative therapies), I would not realign my skull bones a nanometre. Anybody possessing knowledge of human anatomy or a science-based degree should immediately recognise this. Neurosurgeons need drills and saws to move the skull. 

Yet within the Clinical Interest Group for Energy Medicine there is a physiotherapy group promoting the use of craniosacral therapy within the profession. 

I even know of a paediatric physiotherapist who took her child for craniosacral therapy. Apparently, her child's colic (for which there is no known medical intervention) stopped immediately after a few sessions of craniosacral therapy. My own son had colic, and that stopped immediately after a few days of doing nothing.


Critical appraisal of evidence and applied anatomy

For these reasons, I wonder if there is not enough training given to health care professionals to critically appraise evidence. On one thread of the interactive Chartered Society of Physiotherapy's website, set up to argue against the CSP's support of the Energy Medicine Clinical Interest Group and now in its ninth year of debate, one physiotherapist scornfully remarked that science is rubbish anyway - how can a trial be 'controlled' if it is 'randomised'? This suggests a profound lack of training and understanding of basic science methodology. 

Where's that desk...




And despite expertise in advanced and applied anatomy, there are physiotherapists who believe in craniosacral therapy -  that the pressure of a few grams to the skull can regulate the flow of cerebrospinal fluid around the brain. That's an absurd proposition, and proven as such.


Is there any argument for the use of untested therapies?


Elite sports

Leading sports physiotherapist Chris Morgan once discussed with me how elite sport offers time and resources unavailable to many clinicians, such as those working in the NHS. In these cases, he and his medical team can bring to bear perhaps a multitude of evidence-based medical and rehabilitation interventions to maximise recovery. 

But when the athlete is in a race against time to recover from injury for for a one-off event that defines their sporting career - perhaps the European Cup final or the Olympic games - then perhaps every little amount helps, no matter how tiny, to help the athlete recover. The psychological benefits can be as helpful to these athletes as much as any tiny or negligible physical effect. At these times, Chris suggests that “1%ers” could support physiological and psychological recovery of a sportsperson, but stresses they are totally worthless unless they are used only in support of evidence-based treatments.

Against this are the celebrities whose fame seems to lend huge authority to their opinions. 

Jenny McCarthy is a famous proponent of 'alternative' medicine and has been vocal in the argument that autism is caused by vaccines. It isn't, vaccines are harmless and not vaccinating is dangerous. The power of 'celebs' to influence inappropriate health care decisions over and above the advice of experienced scientists and medical professionals is a strange phenomenon of modern society. And it's potentially dangerous.

In the NHS



The potential for harm from untested and unproven 'alternative' therapies has already been mentioned. There are suggestions that Steve Jobs might have survived pancreatic cancer if he had not opted for quackery over the best known treatments of the day.

The NHS is underfunded in so many resources, and physiotherapy teams have been decimated. It is essential that in the limited time physiotherapists have to assess and treat patients they are using evaluations and treatments based on the best evidence.

Physiotherapists in elite sports have more opportunities for implementing 1%ers, for example when the athlete is relaxing, resting from a rehabilitation exercise or receiving a passive treatment. There is also a world of difference between launching an athlete into their career-defining event and raison d'etre, and a postman, painter and decorator, builder etc who returns to a labour-intensive, physically-demanding role with limited paid sick leave.

Private practice

In private practice, physiotherapists are also bound by a professional code. I suggest it is unethical and immoral to take money from (often) desperate patients for unproven and often patently worthless therapies.

What do you think?


Should physiotherapists engage in 'alternative' medicine? 

I could have written pages about various types of nonsense that unethical individuals sell to unwary and ill-prepared members of the public, members of the public who are in despair as they search for a cure to their medical problems. Physiotherapists have specific training and a code of conduct which obligates them to use only interventions with evidence to support them, and certainly not to fleece patients.

As I sit here and write this blog, Horizon (a BBC Science programme) is presenting evidence for healthy eating food fads. This includes a story about a multi-millionaire currently in prison who lied about being a medical doctor and bought his PhD from an online site. He promotes a diet as a cure for cancer. He's made millions of dollars but of course many people have died and the treatment cost them thousands of dollars/pounds. 

What happens when you add Supplements to Complementary and Alternative Medicine?  It reveals SCAM. 



Do physiotherapists have an obligation to advise patients and the public of the evidence for 'alternative' medicine, following objective scientific appraisal of the evidence?

I strongly believe so. But I have suggested that critical thinking and appraisal is beyond some therapists - particularly those who are using 'alternative' medicine in private practice. I have seen use of these therapies defended by the physiotherapist by the cry of 'I have seen the patient get better with my own eyes!'. This supports not the benefits of the 'alternative' medicine but my assertion that these physiotherapists lack critical thinking and objectivity. This has to be addressed.




Finally...

This isn't supposed to be an inflammatory piece, although I accept that it is going to upset some physiotherapists. But I want them to engage in reflective thinking, to consider the evidence and scientific rationale for their 'alternative' therapies, and comment below. But please address my evidence and arguments, and don't resort to attacking me personally. To lighten the mood I leave you with another brilliant comedy take down of 'alternative' medicine. 

I wrote this blog for two reasons, and reiterate that I welcome comments. 
  • I wanted to know what physiotherapists thought of the suggestion they have a responsibility to advise the public on the evidence for, and known benefits and risks of, different health interventions; and 
  • to voice my disquiet and disappointment at the lack of science-based practice and the failure of the profession and HPCP to address some serious limitations in our professional training and conduct. Is the use of 'alternative' medicine against the principles of evidence-based physiotherapy? What are the HPCP for, if not to regulate this? 


Tuesday, 13 September 2016

Issues with measurement in rehabilitation and physiotherapy


"...when you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely in your thoughts advanced to the state of Science, whatever the matter may be." ~ William Thomson, 1st Baron Kelvin (1824 – 1907)

Introduction

In physiotherapy and occupational therapy, the aims of intervention are usually to maximise function - to improve activity and participation in life situations. To evaluate whether intervention achieves these aims, therapists have a large number of measures available to them. 

However, evaluating outcomes is not as straightforward as it may seem. While measures might have been tested for evidence of their validity, reliability and responsiveness, recent developments in psychometric techniques have highlighted limitations in measures developed using older techniques. These limitations have an impact on the accuracy, reliability and validity of the scores themselves, and on the confidence with which we can evaluate the outcome of interventions and of research.

This blog aims to outline the differences between older and more recent psychometric techniques, and outline the common limitations in outcome measures which modern techniques can rectify.


Measurement: less straightforward in rehabilitation

We tend to take the number which outcome measures assign to the quantity we are measuring for granted, at face value. For example, measuring a quality such as weight, length or temperature is generally a straightforward procedure using the appropriate tools - scales for weight, a ruler for length or a thermometer for temperature. These variables are singular, discrete entities and we give little thought to the tools we use to measure them, and we take for granted the numbers associated with the measurement even when our health and welfare depend on them. We trust that the particular tool of measurement in each circumstance is valid i.e. there is good evidence that the scale adequately measures the variable for which it is intended to measure. We trust that the measurements it issues are reliable i.e. that the numbers produced by a scale to represent the quantity of the variable are consistent and reproducible every time, on every occasion and for each different person that uses it. Without this, baking books would be useless, and my Yorkshire Puddings would be very hit-and-miss*. We also trust in the responsiveness of the measure - a change of a degree Celsius in body temperature of a hypothermic or febrile child can be life-threatening so it is essential that the thermometer can capture this change. 

However, in many areas of health care we measure variables that are not so easily to directly measure - mood, anxiety, pain and functional ability, or we rely on subjective feedback. Methods of measuring these human characteristics include the use of questionnaires or checklists.

Typically, measures such as questionnaires include a number of questions, statements or even tasks known as items; each item has a number of response categories, and the respondent endorses the response that most closely indicates his ‘level’ of the variable that the item is measuring. 



It is essential that these measures undergo testing themselves to ensure that they are as valid, reliable and responsive as the thermometer or weighing scales described earlier, as well as meet other essential psychometric properties such as unidimensionality, invariance (the measure is sample-independent), whether the ordinal response categories are working as expected (i.e. endorsed accurately according to the ability of the respondent), no response bias (differential item functioning or DIF),  and for local independence of items (so that the response to one item does not influence the response to another item on the measure). Targeting of the scale - whether the scale possesses floor and ceiling effects, in which a section of the population being measured are off the scale at one or both extremes - should be examined. Unidimensionality is the property that scales actually measure the single characteristic that they are designed and aim to measure, and not a different characteristic altogether. This may seem obvious but one study found a number of measures used to evaluate changes in children's upper limb function were measuring several different traits altogether so that any change in scores was misleading about the intervention's effectiveness.

Traditionally, the reliability and validity of measures have been evaluated using Classical Test Theory (CTT). However, CTT is based on an unsound scientific basis because it is untestable, and is the cause of many limitations now recognised in many measures. This includes wide confidence intervals with the scores, which are always ordinal. More recent psychometric techniques such as the Rasch model offer a number of advantages over CTT. The Rasch model is a mathematical measurement model based on the fundamental scientific principles of measurement. A dataset formed from responses to questionnaire items are evaluated for their fit to the Rasch model. This determines whether the questionnaire meets acceptable standards required for scientific standards of measurement, and allows for the transformation of the ordinal outcome scores to interval level measurement. 

Familiarity with these approaches to psychometric testing is therefore helpful if one is to attempt an evaluation of the essential properties and psychometric standards of the measures.



Developing and evaluating measures


Classical Test Theory

Classical Test Theory (CTT) is based on the theory that the observed score, which is based on the observation of a number of items that relate to the variable being measured, is made up of the true score plus any error. Because the error is random it cancels out leaving the true score. The problem with this theory is that this error is unknown and unobserved, and the evaluation of many psychometric properties in CTT rely on knowing this error.

Item reliability assumes that a good item produces an observed score that is closely related to the true score, and that this observed score varies across individuals (because the variable of interest varies across individuals) and varies across time in one individual e.g. after an intervention (DeVellis, 2006). There is an implication that both true and observed scores covary, and that a measurement of the strength of this covariance will provide an indication of the accuracy of the observed score (DeVellis, 2006). The measurement of the strength of this covariance, and thus the reliability of the measure, is called the correlation coefficient, which if squared gives the proportion of variance shared between the true and observed scores. However, the true score is unobservable, inseparable as it is from the error with which it is associated. CTT holds that a number of assumptions combine to address this problem (DeVellis, 2006). Firstly, the items must be strictly parallel, which is an assumption that comes with its own conditions e.g. items’ covariance with true scores must be equal across items and each item’s error must be independent both of the true score and of every other item’s error (DeVellis, 2006; page S51). This condition is itself unprovable, and therefore CTT is a theory that cannot be “tested, verified, or - more importantly - falsified in any dataset” (page 1098). 

The measure is also only truly validated for the sample on which the psychometric testing was performed i.e. it is sample dependent. 

These are the reasons proposed for the problems with measures described above. For example, because of the unknown qualities and quantities associated with the observed score, only the ordinal raw scores can be used for any statistical operations (non-parametric statistics), and the width of the confidence intervals for individual scores makes conclusions based on change scores between individuals very ambiguous. Also, DeVellis (2006) notes that ordinal scales show varying degrees of responsiveness along the scale from one extreme to the other (by as much as a factor of 15), something which will be illustrated when we explore the Rasch model.  There are other psychometric issues with CTT: the inability to detect response bias (differential item functioning (DIF); and the scales produced are sample dependent. DIF exists when a sub-group within a population endorses responses to items differently to other sub-groups, therefore obtaining a different score even though they have the same amount of the variable being measured. An example of this might be pain or the level of discomfort due to a common cold when measured on a scale given to a group that includes both men and women: one might find that, despite the same level of discomfort, women might endorse items to obtain a lower score for each variable than men.Yes, a man flu detector.

DeVellis (2006) suggests that the nature of CTT-developed measures supports the use of less-appropriate items that contribute less satisfactorily to measurement, stating that “adding items can offset this problem and theoretically, just about any desired level of reliability can be achieved” (page S57). But these advantages do not counter the problems discussed above, and in fact adding items to improve reliability can add problems e.g. local dependency, which causes artificially increased scores.

The requirement in scientific and medical practice for high standards of measurement defines an outcome measure that is valid and reliable, invariant across samples, unidimensional, is free of locally dependent items and is free of DIF. The Rasch model is a mathematical model which offers the opportunity to perform parametric statistical analysis of item responses, thus allowing an objective evaluation of the items’ psychometric properties. Each item can then be modified to address the psychometric deficiencies or, if necessary, deleted altogether. Furthermore, because the Rasch model is based on fundamental scientific principles of measurement, it models the expected behaviour of the questionnaire response dataset for the questionnaire to achieve linear (interval-level) outcome scores.

There is nothing wrong with using ordinal data, and Classical Test Theory can only produce measures with ordinal outcome scores, but it is important to recognize the limitations. For one thing, there are varying confidence intervals of up to 30% with ordinal scores. Another major disadvantage is that the lines which divide the ordinal categories (e.g.Easy, Difficult, Impossible, or No Pain, Dull pain, Sore, Tolerable, Agony) are arbitrary and categories are not necessarily of identical width or quantity, even when that quantity can be more accurately measured e.g. degree classifications. Moreover, a change in scores between two ordinal categories does not mean that the change two different ordinal categories in the same scale is equal. In fact, given the width of the categories, it is likely that sometimes a change will take place that goes unrecognized while in others, a tiny change will be seen as a change in scores. For example, a change of 8% from 61% to 69% would not change the category of a degree classification but a change of 1% from 59% to 60% would. Finally, although non-parametric statistical procedures which use ordinal data are pretty robust, they produce conservative results (because you cannot add, subtract etc - only rank the results in ascending order)  that can affect the clinical and scientific significance of the findings.

Problems with ordinal data

Even senior academics still make pretty basic errors with data, including averaging ordinal data and calculating standard deviations - it's just wrong, very wrong.





The recognition that ordinal data is misused, supports misinferences and is unhelpful to clinicians and researchers is not a recent advance (Merbitz et al., 1989, Stucki et al., 1996). The main problem with the use of ordinal data stems from its very nature, that of grouping similar levels of the variable being measured into different categories that can be ordered by size. The dividing lines between these groups are arbitrary (Merbitz et al., 1989) and each group is not necessarily of identical width or quantity, even when that quantity can be more accurately measured e.g. undergraduate degree classifications. Often, the different groups are given consecutively-numbered identifiers, and this potentially causes the main problem: that of treating each category as being numerically equal. This is not a property of ordinal data. For example, the Manual Ability Classification System is an ordinal classification system with five categories of ascending mobility limitation. Categories, or levels of mobility limitation, are numbered from I – V. The ordinal nature of the GMFCS means that there is increasing mobility limitation as the levels increase in numerical value i.e. Level I describes less mobility limitation than Level II. However, a change from Level I to Level II does not necessarily represent an equal change of mobility as a change from Level II to Level III.

Ordinal data can influence health and research outcomes with this misinference (Merbitz et al., 1989). If a child who was positioned near the top of Level II received an intervention that caused a small improvement in mobility – just enough to take them into Level I, this would indicate success for the intervention; however, the results of an intervention that caused a child’s mobility to progress from the bottom of Level II to the top of Level II would indicate failure of the intervention despite potentially greater improvement in mobility. A clearer example of this problem is illustrated in Figure 3-1 below. This shows the photo finish of two 100 metres sprint events. For the athletes in third place to achieve a silver medal, one of them will have to work somewhat harder to make this transition to the higher ordinal category; furthermore, even if they each run faster by a second, equating possibly to a 10% improvement, they might still not achieve the improvement in ordinal outcome measures if the second place athlete also improves. The impression given by First, Second, Third is of equal intervals, and ordinal data is commonly treated this way. 


It is these unknown and varying distances between and within categories that make ordinal data inappropriate and inaccurate for statistical operations of subtraction, addition, division and multiplication (Merbitz et al., 1989, Svensson, 2001). Yet ordinal outcomes are frequently used in parametric statistical analysis, or presented as means and standard deviations. This problem is highly visible in even common outcome measures of rehabilitation. The outcome score of the Canadian Occupational Performance Measure (COPM) is not only ordinal but is generated by addition and division of ordinal scores to create a mean score.

Hobart and coauthors draw attention to the possibility that the confidence interval of an ordinal outcome score for individuals can occupy 30% of the available range of the scale, suggesting that a remarkable change in scores will be necessary before any significance can be attached to it. This has potential implications for the change in COPM scores that is regarded as clinically significant – a change of 2 out of 10. Hobart et al. (2007) further suggest that that ordinal scales are by their nature likely to be unresponsive. This could result in type-II errors. Arguments against more responsive scales include the likelihood of type-I errors but this is easily avoided by establishing a change score for the measure which indicates clinically significant change. Common sense suggests that for a measure to detect a clinically significant change in the characteristic being measured, its sensitivity and scale calibrations must be greater than that of the minimally significant change.


If ordinal-level outcome scores are unacceptable for health research, a psychometric model for establishing the fundamental properties for scientific measurement and interval-level measurement must be used. The Rasch model provides the opportunity to develop measures which meet these properties and to test for them in existing measures.


An outline of the Rasch Model

The underlying principle for the Rasch model is that being able to achieve a task, or pass an item (Yes or No), is influenced by both the difficulty of the task or item and the ability of the person. That is, the probability of a person endorsing, or ‘passing’, an item depends on how hard the item is and how much ability they have, and the hardest item on a measure will be endorsed or passed only by those with the greatest ability of the trait being measured. Endorsing or passing an item illustrates a specific ‘quantity’ of the trait being measured; therefore it is probable that all easier items will be passed or endorsed by that person, and also probable that items of more difficulty will not be endorsed. The Rasch model therefore resembles a probabilistic form of Guttman scaling.

Guttman scaling usually forms a deterministic pattern, as shown in the hypothetical example below. Items of increasing difficulty are passed up to a point where items become too difficult. This point varies depending on the ability of the person and how hard the item is. 

Guttman pattern

The Rasch probabilistic model of the Guttman scaling, then, is less strict than the deterministic model and is the model against which the dataset from item responses of the measure undergoing psychometric testing is evaluated. This allows that, within a narrow border there is room for error. Outside of this narrow border, which is assessed for goodness of fit by a chi-squared test, items that do not fit do not meet the requirements for scientific measurement demanded by the Rasch model. 

There is another important consideration, however - that of central tendency, here shown as the way that ability tends to cluster as a mean, with fewer people (of very poor and very high ability) at the extremes, and the greatest number of people (of average ability) in the centre. This is what produces the well known bell curve of a Normal distribution and gives us an ogive rather than the straight line shown in the Guttman scale example. 



Here we can see how the ordinal scale (along the y-axis) can vary in responsiveness at different parts of the scale: a change of item difficulty scores from ten to twenty units and from a score of 40 to 50 units (i.e. a change of ten units for each) gives a change of person ability on a linear scale that differs by nearly 50% (i.e. from -3 to -1.6 and from -0.2 to 0.5 respectively), illustrating one of the problems with ordinal scores. 

If the item responses show a good fit to the Rasch model the outcome measure that produced them is determined to have met the fundamental principles of measurement for achieving linear (interval level) outcome scores. Before removing misfitting items, though, it is essential to review the items - it is possible that a change in wording, or modifying the item activity, will bring the item within acceptable fit to the model. The analysis also evaluates person fit, and gives a strong statistical analysis of the targeting of the scale to the population and trait being measured, and whether the scale is targeting at a calibration to differentiate between individuals of different but similar ability. It is important to note that although Rasch analysis can identify psychometric problems with the items, and give guidance to how to address them, it is essential to give full attention and consideration when developing the items and their response categories - no amount of testing will overcome poor items.  



Software used for performing statistical procedures to evaluate fit to the Rasch model (Rasch analysis) also tests a number of other measurement issues, all of which we have discussed above and are necessary for permitting transformation to interval outcome scores, and opens up the use of Rasch analysis to the wider community.  It tests the internal construct validity of the measure,  unidimensionality, the invariance of items (the measure is sample-independent), evaluates whether the ordinal response categories are working as expected (i.e. endorsed accurately according to the ability of the respondent), response bias (differential item functioning or DIF),  and for local independence of items (when the response to one item influences the response to another item on the measure). This response dependency was mentioned as a problem with CTT. Finally, a Rasch analysis will evaluate the targeting of the scale: whether the scale possesses floor and ceiling effects, in which a section of the population being measured are off the scale at one or both extremes.


Other issues

Two other issues are important when evaluating the measures  you are using. Some measures are developed to measure a trait across a broad range of conditions (generic measures). Beware that these are likely to be unresponsive to change. 


Round up



This, my first ever blog, describes common limitations with outcome measures. All measure should have evidence of their reliability, validity, responsiveness, unidimensionality, invariance, absence of local dependence and response bias; floor and ceiling effects should be described if they have not been addressed. There are compelling arguments against the use of ordinal outcome scores, which have wide confidence intervals and which vary hugely in their responsiveness along the scale.

Item Response Theory in the form of Rasch analysis has demonstrable advantages over Classical Test Theory for developing and testing outcome measures, such as determining that the outcome measure meets the fundamental principles of scientific measurement that permit transformation of the ordinal outcome scores into linear (interval level) measurement.

* (For Yorkshire puddings, the key factors are: pre-heating the oil; keeping the oil hot on the cooker while adding the batter; and leaving the oven door shut for at least 15 mins after they go in). You're welcome.