A valid and reliable method of evaluating the delivery of manual-based psychosocial treatments for alcohol dependence and misuse

Alcohol Insight Number 33

Research and Development Grant

Introduction

The United Kingdom Alcohol Treatment Trial (UKATT) research team have devel­oped a Process Rating Scale (PRS) to measure the delivery of Motivational En­hancement Therapy (MET) and Social Behaviour and Network Therapy (SBNT) for the treatment of alcohol dependence and misuse in the UK context. This method is unique in its ability to rate the delivery of SBNT, and to compare this treatment with another treatment of proven effectiveness.

Social Behaviour and Network Therapy aims to identify and work with a social net­work supportive of positive change in eight structured sessions including network based communication and coping skills, relapse prevention and alternative ac­tivities to drinking. The purpose of Motivational Enhancement Therapy is to bring about an internally motivated decision to change drinking behaviour.

The purpose of the present study was to develop and validate a manual based method of rating treatment fidelity, based upon frequency and quality of the de­livery of treatment components, treatment manual adherence, therapeutic style and discriminability between treatments. The UK Alcohol Treatment Trial Process Rating Scale (UKATT-PRS) is a 26 point rating scale designed to be time-efficient, to allow all sessions to be rated by the same rater, to be readily adaptable to use with a range of therapeutic approaches to substance misuse treatment and ap­propriate for use in a UK context.

For each item, a definition, a description of the characteristics of high and low ratings for frequency and quality and examples of therapist dialogue illustrating these were provided in manual form. General guidance on differentiating the fre­quency and quality of therapist behaviours, on avoiding common pitfalls relating to possible rater bias and on the method for note-taking during the session was also included.

Video recordings of two psychosocial treatments of alcohol misuse and depend­ence delivered in the UK Alcohol Treatment Trial (UKATT) were available for the purpose of validating the rating scale. One video per client (where available) was sampled for process rating. The sample of over 400 video tapes was stratified by treatment (MET, SBNT), session number (1-3 for MET, 1-8 for SBNT) and centre (South Wales, Leeds, West Midlands). Replacement sampling was used when a video was subsequently found to be un rateable, in order to retain the balance be­tween treatments, session numbers and centres. A target of 50 randomly selected videos was set for double rating and of these 25 would be triple rated by a further two independent raters. This method ensured inclusion of ratings throughout the entire treatment phase of the trial and ensured balance by treatment, session number and centre.

Findings

The scale was able accurately to detect components of each of the treatments and to discriminate between them, to demonstrate that treatment was delivered as planned and that characteristics of the other treatment were either missing or were delivered infrequently. The summary ratings for each treatment are indica­tive of the ability of the scale to discriminate the two treatments. All treatment-­specific items in the UKATT-PRS showed significant differences between the two treatments indicating that the scale is able both to discriminate the treatments generally, and to detect the delivery of all specific components of the content and style of the two treatments that are included in the scale.

Measurement of agreement between the three independent raters for individual items was relatively high and agreement was comparable for MET and SBNT ses­sions. Agreement between independent raters was equally high for frequency and quality ratings. This suggests that the scale is a reliable measure of components of the two treatments.

Implications

The requirement to monitor the delivery of psychological and social treatment underlies clinical governance of routine clinical practice, supervision and psycho­therapy research. It is therefore necessary to have a reliable and valid method to quantify and assess the quality of treatment delivery. Treatment integrity or fidelity checks provide the means to examine the extent to which treatments are delivered and the quality of such delivery.

Interpretation of the findings in both effectiveness and efficacy trials in psycho­therapy research requires measurement of treatment implementation. Variations in competence can be identified and potential treatment effects more accurately attributed. For example, where there are no treatment fidelity checks, treatment effects could be wrongly attributed to the treatments themselves rather than a difference in therapist competence. Equally the potential emergence of a treat­ment effect may be masked by variations in the extent and quality of the delivery of treatment.

The UKATI-PRS is a valid and reliable method of rating the delivery of two psy­chosocial treatments for alcohol problems and dependence. It is likely to be able to be adapted for the purpose of rating the delivery of other psycho-social treat­ments applying the same principles used in its development. It can therefore form the basis of measuring performance and treatment fidelity in clinical trials, in treatment audit and in routine supervision of practice.

Research Team

Tober, G., Clyne, W., Finnegan, O., Farrin, A. and Russell, I. in collaboration with the UKATT Research Team

Downloads

This Alcohol Insight
The Final Report

Tober, G., Clyne, W., Finnegan, O., Farrin, A. and Russell, I. in collaboration
with the UKATT Research Team