Digital blinding of radiographs to mask allocation in a randomized control trial

Abstract

AIM

To demonstrate the effectiveness of a digital radiographic altering technique in concealing treatment allocation to blind outcome assessment of distal femur fracture fixation.

METHODS

Digital postoperative anteroposterior and lateral radiographs from a sample of 33 randomly-selected patients with extra-articular distal femur fractures treated by surgical fixation at a Level 1 trauma center were included. Using commercially available digital altering software, we devised a technique to blind the radiographs by overlaying black boxes over the implant hardware while preserving an exposed fracture site for assessment of fracture healing. Three fellowship-trained surgeons evaluated a set of blinded radiographs twice and a control set of unblinded radiographs once. Each set of radiographs were reviewed independently and in a randomly-assigned order. The degrees of agreement and disagreement among evaluators in identifying implant type while reviewing both blinded and unblinded radiographs were assessed using the Bang Blinding Index and James Blinding Index. The degree of agreement in fracture union was assessed using kappa statistics.

RESULTS

The assessment of blinded radiographs with both the Bang Blinding Index (BBI) and James Blinding Index (JBI) demonstrated a low degree of evaluator success at identifying implant type (Mean BBI, far cortical locking: -0.03, SD: 0.04; Mean BBI, standard screw: 0, SD: 0; JBI: 0.98, SD: 0), suggesting near perfect blinding. The assessment of unblinded radiographs with both blinding indices demonstrated a high degree of evaluator success at identifying implant type (Mean BBI, far cortical locking: 0.89, SD: 0.19; Mean BBI, standard screw: 0.87, SD: 0.04; JBI: 0.26, SD: 0.12), as expected. There was moderate agreement with regard to assessment of fracture union among the evaluators in both the blinded (Kappa: 0.38, 95%CI: 0.25-0.52) and unblinded (Kappa: 0.35, 95%CI: 0.25-0.45) arms of the study. There was no statistically significant difference in fracture union agreement between the blinded and unblinded groups.

CONCLUSION

The digital blinding technique successfully masked the surgeons to the type of implant used for surgical treatment of distal femur fractures but did not interfere with the surgeons’ ability to reliably evaluate radiographic healing at the fracture site.

Key Words: Methods, Randomized controlled trials, Patient outcome assessment, Fracture healing, Femoral fractures

Core tip: The purpose of this study was to demonstrate the effectiveness of a digital blinding protocol to conceal treatment allocation and permit blinded assessment of radiographic healing of various distal femur fractures. Digital postoperative radiographs from a randomly-selected sample were blinded using digital altering software and evaluated by three fellowship-trained surgeons. This study demonstrates the success with which an uncomplicated and reproducible technique can blind radiographs of distal femur fractures. The blinding protocol successfully masked the surgeons to the type of fixation devices implanted but did not interfere with reliable evaluation of radiographic union.

INTRODUCTION

Locking plates with standard locking screws are currently used as the gold standard of treatment to stabilize displaced extra-articular distal femur fractures. These plate and screw constructs provide excellent strength to withstand early joint motion and rehabilitation. However, emerging evidence suggests that the rigidity of these constructs may cause complications such as delayed union, nonunion, implant failure, late loss of alignment, and subsequent need for an additional surgical procedure[1-7]. In response to the increasing clinical concern that plates with standard locking screws are too stiff, “far cortical locking” (FCL) screw technology has been introduced to permit controlled micro-motion within a locked construct, which in theory leads to earlier and more predictable healing[8,9]. While initial reports in animal studies and small retrospective clinical series have examined the efficacy of FCL screw technology in treating femur fractures, with more uniform callus formation[6,9], prospective clinical trials comparing standard locking and FCL technologies are necessary to guide treatment recommendations.

The comparison of standard locking screws vs FCL screws is currently being conducted in a multi-center randomized control trial (RCT). One of the challenges of surgical RCT’s is blinding to reduce bias, since surgeons cannot be blinded to treatment allocation. However, blinding independent outcome assessors can be and effective way to reduce bias[10]. The purpose of this study is to demonstrate the effectiveness of a digital blinding protocol to conceal treatment allocation and permit blinded assessment of radiographic healing of distal femur fractures treated with standard locking screws and FCL screws.

MATERIALS AND METHODS

Blinding

Digital postoperative anteroposterior and lateral radiographs from a sample of 33 randomly-selected patients with extra-articular distal femur fractures (OTA 33-A) were included. All patients were treated with a pre-contoured distal femur plate. Twelve subjects had been treated with standard locking screws, while the other 21 had been managed with FCL screws. Radiographs from the 6 wk, 12 wk, and 24 wk postoperative follow-up visits were acquired for each patient. Week 6 images were not always available, and when this was the case, week 2 postoperative images were substituted. All available radiographs from each patient were of adequate quality for use in the study.

Using commercially available digital altering software (Acorn 1.5.5, Flying Meat, Inc., Everett, WA, United States), we devised a technique to blind the radiographs by overlaying black boxes over the hardware while preserving an exposed fracture site for assessment of healing. The radiographs of standard locking screw constructs contained an average of 6 shaft screws, while those of FCL screw constructs contained an average of 4. To deal with this discrepancy, we standardized each radiograph by placing a minimum of 6 blinding boxes on the shaft portion of the distal femur (Figure 1). A single investigator (L.S.) digitally modified the entire series of images. This investigator had minimal experience with digital photo alteration and no previous experience with the digital altering software.

Figure 1 Examples of radiographs before (top) and after (bottom) blinding. Images were taken at 12- (left) and 24-wk (right) post-operation.

Study design

Three attending physicians with fellowship training in orthopaedic traumatology evaluated the images. All of the evaluators had experience operating with both types of locking screws and interpreting postoperative radiographs. All were aware of, and one was directly involved with, the development of the blinding protocol.

The 33 patients were each assigned a subject number, and random number generating software was used to order the radiographs corresponding to each patient into two PowerPoint (Microsoft, Redmond, WA) presentations, one with unblinded images and the other with blinded images. Each PowerPoint slide contained orthogonal views of four radiographs corresponding to each patient. One set of anteroposterior and lateral views from the time point of healing assessment-either the 12-wk or 24-wk postoperative follow-up visit-and another set of views from a previous time point was used for comparison. Specifically, week 24 images were paired with week 12 images, and week 12 images were paired with week 6 images.

The blinded set of radiographs was assessed twice and the unblinded control once. The evaluations were performed independently and in a randomly-assigned order. A minimum of 2 wk elapsed between evaluations.

For each PowerPoint slide (unblinded and blinded radiographs), the surgeons assessed: (1) the type of hardware used (standard locking screws, FCL screws, or unsure); and (2) status of bony union (healed, not healed, or unsure). Although radiographic fracture healing is subjective and without a clear gold standard[11], in this study radiographic healing was defined as evidence of minimum bridging of 2 cortices. For the blinded images, the evaluators were also asked whether or not the blinding interfered with his or her ability to assess fracture healing (yes, no). If the blinding did interfere, the evaluators were asked to specify which area of the image was problematic (proximal, distal, or both).

Statistical analysis

Statistical analysis was performed using Stata 10.0 (Stata Corp LP, College Station, TX) to calculate the Bang Blinding Index[12,13] and the James Blinding Index[14]. The Bang Blinding Index is commonly used to measure the degree of agreement between evaluators beyond the degree expected by chance. Scores range from -1 to 1, with 1 representing a complete lack of blinding, 0 representing perfect blinding, and -1 representing opposite guessing, which may be related to unblinding[12]. The James Blinding Index measures the degree of disagreement between evaluators. Scores range from 0 to 1, with 0 representing a complete lack of blinding, 0.5 representing completely random blinding, and 1 representing perfect blinding[15].

The kappa statistic is related to the James Blinding Index and was calculated to measure the degree of fracture union agreement between evaluators. The kappa statistic was interpreted using the methods of Landis and Koch, commonly used for interpreting inter-evaluator agreement for qualitative or categorical outcome measures[16].

RESULTS

Evaluation of blinding

The assessment of blinded radiographs with both the Bang Blinding Index and James Blinding Index demonstrated a low degree of evaluator success at identifying implant type, suggesting near perfect blinding (Table 1). The mean Bang Blinding Index was 0 ± 0 for the images with standard locking screws and -0.03 ± 0.04 for those with FCL screws. The mean James Blinding Index was 0.98 ± 0.

Table 1 Blinding assessment with blinded radiographs.

	BBI-FCL screw	BBI-standard screws	James Blinding Index
Observer 1	0	0	0.99
Observer 2	0	0	0.99
Observer 3	-0.08	0	0.98
Mean	-0.03	0	0.98
SD	0.04	0	0

BBI: Bang Blinding Index; FCL: Far cortical locking; SD: Standard deviation.

The assessment of unblinded radiographs with both indices demonstrated a high degree of evaluator success at identifying implant type (Table 2), as expected. The mean Bang Blinding Index was 0.92 ± 0.87 for the images with standard locking screws and 0.89 ± 0.19 for those with FCL screws. The mean James Blinding Index was 0.26 ± 0.12.

Table 2 Blinding assessment with unblinded radiographs.

	BBI-FCL screw	BBI-standard screws	James blinding index
Observer 1	1	0.85	0.32
Observer 2	1	0.85	0.32
Observer 3	0.67	0.92	0.12
Mean	0.89	0.87	0.26
SD	0.19	0.04	0.12

BBI: Bang Blinding Index; FCL: Far cortical locking; SD: Standard deviation.

Evaluation of fracture healing

There was moderate agreement with regard to assessment of fracture healing among the evaluators in both the blinded and unblinded arms of the study (Table 3). There was no difference in agreement between the blinded and unblinded groups.

Table 3 Agreement scores for the assessment of fracture healing.

Type of review	Kappa	95%CI
Blinded	0.376	0.253-0.515
Unblinded	0.353	0.252-0.453

DISCUSSION

This study demonstrates the success with which an uncomplicated and reproducible technique can blind radiographs of distal femur fractures. The blinding protocol successfully masked the surgeons to the type of locking screws implanted in the distal femur. Statistical analysis with the Bang Blinding Index and James Index scores confirmed the success of the blinding protocol: The interval estimates were all close to 0 and 1, respectively, representing near perfect blinding. The blinding protocol did not interfere with the surgeons’ ability to evaluate radiographic healing at the fracture site. There was no difference in agreement for assessment of fracture union between the blinded and unblinded radiographs. Furthermore, moderate agreement of fracture healing using radiographs is consistent with previously published literature[17].

The protocol used to blind the type of hardware placed in the distal femur is based upon the previously published work by Karanicolas et al[18]. Karanicolas et al[18] identified three different methods of digitally concealing radiographic hardware in the femoral neck: The “blackout” technique involves the placement of an opaque polygon over the hardware, the “subtraction” technique involves digitally copying bone from another region and passing it over the hardware, and the “overlay” technique involves digitally copying one implant and passing it over a radiograph consisting of the other implant. Although all three techniques successfully blinded evaluators to the type of hardware implanted, the “blackout” technique resulted in the most difficulty in identifying hardware and required the least amount of time per radiograph. Although our study differs in the anatomic location of implanted hardware, focusing on the distal femur, our digital blinding protocol is similar to their most easily reproducible and effective method, the “blackout” technique.

Our results must be interpreted within the limitations of the study design. The number of fractures included in this study was limited, with radiographs from only 33 subjects being assessed. Furthermore, the quality and profile of the radiographs were not uniformly standardized, potentially affecting the radiographic assessment. Finally, we blinded the distal articular portion of the plate to mask subtle differences in the plate design of the various manufacturers. This may have impeded some assessments of fracture healing; however, this will not be necessary in the current RCT because all enrolled patients will receive the same locking plate.

Our study design has a number of strengths, including the separation of a randomized order of independent images into two PowerPoint presentation modules (unblinded and blinded) for individual surgeon assessment a minimum of 2 wk apart. All radiographs were evaluated for distal femur fracture union by the overall impression of trauma surgeons, which has been reported to be a moderately reliable method of assessing the quality of radiographic healing of femur fractures[19]. Additionally, our blinding protocol was successfully utilized without negative consequence on fracture healing assessment. A gathering of results and thorough statistical analysis was performed independently to test the criteria for effective blinding, further limiting detection bias. The James and Bang Blinding Indices were both used to limit the effect of “correct guessing”[5-7]. Finally, our blinding method is effective, efficient, and easily reproducible for future study designs.

Distal femur fractures are commonly treated by plate and screw constructs, but comparative efficacy research in this field is difficult to perform due to the practical limitation of blinding outcome assessments. This study provides not only a simple blinding technique for outcome evaluation but also a method to assess the success of blinding, both of which increase the validity of future trials which compare standard locking screws and FCL screws in the treatment of distal femur fractures. These techniques may be applied to investigations in other fields of orthopaedic surgery which involve evaluation of radiographs containing opaque implants.

COMMENTS

Background

Surgical fixation of distal femur fractures with locking plates and far cortical locking screw (FCL) technology may cause controlled, micro-motion at the fracture site to allow more reliable and uniform callus formation for predictable healing. However, no comprehensive studies comparing the effectiveness of FCL technology to standard locking screws in the treatment of distal femur fractures exist. The aim of this study was to devise and analyze a digital radiographic altering technique to conceal treatment allocation and blind outcome assessment of distal femur fracture fixation. This would allow unbiased comparison of distal femur fixation methods.

Research frontiers

Locking screws are necessary for surgical fixation of displaced extra-articular distal femur fractures. Standard locking or FCL screws may be used. However, there are few studies which directly compare, without bias, standard locking and FCL screws in effectively healing distal femur fractures.

Innovations and breakthroughs

The authors created and analyzed a digital blinding technique to objectively assess radiographic union of distal femur fractures treated with two types on plate-and-screw constructs. Statistical analysis with the Bang Blinding Index and James Index scores confirmed the success of the blinding protocol. There was no statistically significant difference in agreement for assessment of fracture union between the blinded and unblinded radiographs.

Applications

The results of this study suggest that a simple digital radiographic blinding technique may be a reliable method for objective, unbiased outcome evaluation in trials comparing the efficacy of standard locking screws and FCL screws in the treatment of distal femur fractures. These techniques may be applied to investigations in other fields of orthopaedic surgery which involve evaluation of radiographs containing opaque implants.

Terminology

Digitally altering radiographs with black boxes overlaying hardware, while preserving an exposed fracture site, is a technique to blind outcome assessors in evaluating distal femur fracture fixation.

Peer-review

This is an interesting study on the use of a digital protocol to blind outcome assessors in evaluating radiographic union of bone fractures after surgical fixation.