Discussion
Using data from two clinical trials of rTMS for TRD, we evaluated five different crosswalk models for converting scores between HRSD and MADRS. We have developed practical conversion techniques for these scores at baseline, post-treatment and follow-up time points as well as for the changes in scores between baseline and post-rTMS treatment. All models developed using the current dataset performed similarly in terms of RMSE when developed and applied at the same time point; however, the models diverged in terms of their performance at differing time points and also with respect to their underlying assumptions. The RF model was consistently the worst-performing model at all time points. Linear regression, SVR and rTMS equipercentile model performed similarly. However, when examining longitudinal trajectories using models developed with baseline data, the rTMS equipercentile model demonstrated the least amount of systematic bias. In the SVR model, using individual items instead of sum scores or adding patient characteristics as covariates did not improve model performance. Collectively, these results suggest that the rTMS equipercentile model may be the optimal model as it was the most robust and least biased approach for converting between MADRS and HRSD among individuals with TRD receiving rTMS. Additionally, the distribution of the input data needs to be considered when developing crosswalk models. Differences in the score range (eg, limited score ranges of the baseline due to the eligibility threshold) may lead to inaccurate models. We have also made the implementation of these models readily available by creating an online interactive tool to facilitate the conversion between HRSD and MADRS for rTMS clinical trial data.
In the context of prior work, we identified several issues regarding the use of the pharmacotherapy equipercentile model8 for crosswalking between MADRS and HRSD among individuals receiving rTMS treatment. Systematic differences were revealed between the pharmacotherapy equipercentile model’s predicted scores and the actual scores—MADRS estimates were underestimated, while HRSD estimates were overestimated. This may be due to several reasons. First, like most modern rTMS trials, the two trials included in the present study only enrolled patients with TRD,21 who present with different clinical features, such as symptom severity, chronicity and presence of comorbidity, as compared with non-TRD patients. Second, antidepressant medications and rTMS may correspond to distinct treatment response variability. Prior studies have identified different treatment response trajectories in antidepressant and rTMS trials.22 23 After treatment, such differences in response trajectories may lead to different conversions between HRSD and MADRS. This leads to the possibility that converting MADRS and HRSD scores may require different crosswalk models for different treatment modalities. However, confirming this possibility will require external validation of the internally developed models (ie, the rTMS equipercentile model). Even though the present results have highlighted the differences between crosswalking models for different treatment modalities, the risks of overinterpretation remain. New crosswalking models could be tailored for numerous subgroups of patients, such as those receiving specific rTMS protocols or those with specific clinical features. However, researchers must be cautious, for subgroup analysis could be misleading.24 Existing theoretical and empirical evidence must be considered before such studies are conducted. Furthermore, the external validity of the recommended models (ie, rTMS equipercentile model) needs to be assessed before they can be routinely recommended.
While the rTMS equipercentile model in this work appeared to be optimal and is the approach most commonly used for psychometric scale conversions,14 15 the SVR model, like many machine-learning approaches, is promising due to its ability to detect complex and non-linear patterns and high predictive accuracy when trained with ‘big data’.19 The failure of this study to find the superior performance of machine learning approaches in contrast to other work17 may be due to insufficient sample size needed to take advantage of the strengths of machine learning approaches. The linear regression model was included in the current study because it has been widely used in almost all disciplines of psychiatry and is easy to interpret. Although it achieved good performance when crosswalking between HRSD and MADRS using data at the corresponding time points, it was not favoured for two reasons. First, the performance of the linear regression dropped dramatically when using the baseline model to predict scores from other time points, demonstrating poor robustness. Second, its theoretical assumption of the linear relationship between HRSD and MADRS, the normal distributed error terms with uniform variance, may not hold, given the nature of these scales. Nevertheless, the performance of this model still provided valuable information when compared with other models. Additionally, one possible reason for the linear regression model’s relatively good performance is the multiple imputation method used in the current study, which involved a regression model to predict the missing values according to the existing data. This might generate a dataset that favours a linear regression model, especially when the missingness is substantial (online supplemental table S1).
One notable strength of the present research is including an rTMS clinical trial in a group of older adults with TRD.7 Depression in older adults is common but characterised by unique clinical features as compared with MDD in younger adults.25 Emerging evidence has shown that rTMS is an effective treatment option for depression in older adults.26 The tool developed in the current work will facilitate comparing study results among existing rTMS clinical trials in the older adult population. While the present study does not have a direct clinical impact, it can facilitate future studies—such as meta-analyses like the one by Kan et al,27 which aggregated clinical outcomes from all available rTMS clinical trials and analysed the effect sizes of rTMS on different symptoms including depressive symptoms—generating clinically relevant knowledge by enabling the aggregation of multiple clinical trials using differing outcome measures. The present study, therefore, provides a tool to advance clinical knowledge by enabling analyses requiring large-scale datasets, such as analyses assessing the heterogeneity of treatment effects.28
There are some limitations regarding the present research to consider. Although in the context of psychiatric clinical trials, the sample size of the present study was quite large,7 18 in the field of machine learning, it is relatively small.29 Relatedly, the precision of the present models needs to be considered with caution as the sample size of the study used to develop the existing pharmacotherapy equipercentile model was notably larger (N=4388) than the present study (N=380), which has resulting implications for the precision of the current model outputs. Another limitation of this study is that we only considered two clinician-rated scales. Due to limited data availability, we also conducted all model assessments using internal validation procedures rather than external validation. Future work should perform external model validation using rTMS clinical trial data from separate data sources. It should also be noted that the present findings pertain specifically to TMS targeting the DLPFC.30 Future crosswalk studies in clinical trials with larger sample sizes and using different modalities (eg, different rTMS protocols and targets) will, therefore, be an important area of future work. The same rater administered the HRSD and the MADRS according to the design of the included clinical trials.7 18 While such practice had ensured the reliability between these two scales and facilitated the comparison, it led to unavoidable dependency between HRSD and MADRS scores. The differences regarding the ranges of scores at different time points and score changes must be considered when applying the crosswalking models. Models’ performances tend to drop when applied to a new dataset with different score ranges, so it is recommended to use the ‘current’ model and not to convert scores that are out of the typical score ranges at the given time point or of the score changes.
In this study, we leveraged data from two large rTMS clinical trials to evaluate the performance of five crosswalk models for converting between HRSD and MADRS. We found that the rTMS equipercentile model developed using the trial data was the best-performing and most robust model. However, the SVR model performed well and demonstrated significant potential for future work. Finally, we have developed an interactive online tool for converting HRSD/MADRS scores, which will be useful in aggregating existing clinical data to facilitate evidence synthesis.
