Retrospective Evaluation of Various Serological Assays and Multiple Parameters for Optimal Diagnosis of Lyme Neuroborreliosis in a Routine Clinical Setting

In total, 150 (13.7%) of 1,098 consecutive patients who underwent at least one LP in the predefined study period were included in the current study. Six additional LNB patients were included from outside the predefined study period, all of whom had taken part in two other studies (43, 44). Details on the selection of the 156 patients have already been published, these patients have also been used to evaluate two commercial CSF-CXCL13 assays (45).

All patients were classified as definite LNB, possible LNB, or non-LNB patient based on the EFNS guidelines (3) and consensus strategy according to the flowchart in Fig. S1. Details with regard to clinical symptoms suggestive for LNB, an alternative diagnosis that ruled out LNB, and test results (i.e., pleocytosis and intrathecal Borrelia-specific antibody synthesis determined by the CSF-serum assays) needed for the classification of patients are shown in Table S2. The number of possible and definite LNB patients in this study differed slightly from that in our previous study (45), as intrathecal Borrelia-specific antibody synthesis was based on either a consensus strategy (this study) or the IDEIA results (previous study).

Ten (6.4%) of the 156 patients were classified as definite LNB patient, and 7 (4.5%) of the 156 patients were classified as possible LNB patient (Table 1). Of the seven possible LNB patients, four (57.1%) had pleocytosis and three (42.9%) had intrathecal Borrelia-specific antibody synthesis according to the consensus strategy. A total of 139 (89.1%) of the 156 patients were classified as non-LNB patient. Thirty-nine (28.1%) of them had clinical symptoms suggestive of LNB, of whom 36 (92.3%) had neither pleocytosis nor intrathecal Borrelia-specific antibody synthesis according to the consensus strategy and 3 (7.7%) had pleocytosis only. For these three patients, another cause for their symptoms was found (Table S2). One hundred (71.9%) of the 139 patients classified as non-LNB patient did not have clinical symptoms suggestive of LNB. Eighty-one of them had neither pleocytosis nor intrathecal Borrelia-specific antibody synthesis according to the consensus strategy, and the remaining 19 had pleocytosis only.

Table 2 shows a detailed overview of the demographic characteristics and clinical parameters among the three study groups. Pleocytosis, an elevated total protein level in CSF, and a positive Borrelia species PCR result on CSF was found among definite LNB patients more often than among possible LNB and non-LNB patients, and this was significantly more often than among non-LNB patients. CSF-CXCL13 positivity was found among definite LNB patients significantly more often than among possible LNB and non-LNB patients. No significant differences were observed among the three study groups with regard to gender, age, symptom duration, and CSF-glucose levels.

All definite and possible LNB patients had a positive (or equivocal) C6 enzyme-linked immunosorbent assay (ELISA) result on serum, and this was significantly higher than that among non-LNB patients (Table 2). Two-tier serology results on serum showed that the percentage of positive test results for Borrelia-specific immunoglobulin (Ig)M, IgG, and the combined IgM and IgG results, hereafter referred to as overall Ig results, was comparable between definite and possible LNB patients. These percentages were significantly higher among those two patient groups than among non-LNB patients, except for the percentage of positive test results for Borrelia-specific IgM between definite LNB and non-LNB patients.

Measurement of CSF and serum albumin concentrations and the subsequent calculation of the CSF/serum quotient for albumin (Q Alb) provided insight into the functionality of the blood-CSF barrier. A dysfunctional blood-CSF barrier was found among definite LNB patients more often than among possible LNB and non-LNB patients, and this was significantly more often than among non-LNB patients (Table 2).

Intrathecal total antibody synthesis among definite LNB patients was based on an IgM with or without an IgG response (Table S2). Intrathecal total antibody synthesis among possible LNB patients was based on a solitary IgM response and among non-LNB patients based on an IgM and/or IgG response. Intrathecal total IgM synthesis was found among definite and possible LNB patients significantly more often than among non-LNB patients (Table 2). Intrathecal synthesis of total IgG with or without total IgM was more common among definite LNB patients than among possible LNB and non-LNB patients, although this was significant only between definite LNB and non-LNB patients.

For each of the five CSF-serum assays, the percentage of positive AI results for Borrelia-specific IgM, IgG, or both among definite LNB patients was significantly higher than that among non-LNB patients (Table 3). Most of these percentages were also higher among possible LNB than among non-LNB patients, and this was significant for Borrelia-specific IgG and overall Ig, except when the IDEIA was used. For Borrelia-specific IgM, however, this was significant only using the Enzygnost IgM ELISA.

For all definite LNB patients, and for all possible LNB patients without pleocytosis (n = 3), a positive Borrelia-specific AI result was based on an IgG response with or without IgM (Table S2). Of the possible LNB patients with pleocytosis (n = 4), two had a positive Borrelia-specific AI result for IgM only in either one or two CSF-serum assays. Twelve (8.6%) of the 139 non-LNB patients had a positive Borrelia-specific AI result for IgM and/or IgG in a minority of the CSF-serum assays (Table S2).

All definite and most of the possible LNB patients had a positive C6 ELISA result on CSF, and this C6 ELISA positivity was significantly higher than that among non-LNB patients (Table 3). Using the recomLine immunoblot (IB) on CSF, IgM was not detected at all. RecomLine IgG IB positivity using the revised interpretation criteria of the manufacturer (Table S3), which were elaborated on throughout this article, was more common among definite LNB patients than among possible LNB and non-LNB patients, although this was significant only between definite LNB and non-LNB patients (Table 3).

Detection of Borrelia-specific antibodies in CSF is no direct proof that these antibodies are intrathecally produced and can also be explained by passive diffusion from the blood. C6 ELISA positivity in CSF among definite LNB patients did not differ from that in serum (100% [10/10], both) (Table S2). C6 ELISA positivity in CSF among possible LNB patients was comparable to that in serum (85.7% [6/7] and 100% [7/7], respectively) (P = 1.000). Among non-LNB patients, however, C6 ELISA positivity in CSF was significantly lower than that in serum (3.6% [5/139] and 27.3% [38/139], respectively) (P < 0.001; false-discovery rate [FDR] ≤ 2.0%).

The overall Ig recomLine IB positivity in CSF among definite LNB patients was comparable to that in serum (80.0% [8/10] and 90% [9/10], respectively) (Table S2) (P = 1.000). Among possible LNB patients, the overall Ig recomLine IB positivity in CSF was lower than that in serum (14.3% [1/7] and 85.7% [6/7], respectively) (P = 0.074). Among non-LNB patients, the overall Ig recomLine IB positivity in CSF was significantly lower than that in serum (3.6% [5/139] and 25.2% [35/139], respectively) (P < 0.001; FDR ≤ 2.0%).

The sensitivities of the IgM assays to diagnose LNB among definite and possible LNB patients ranged from 0.0% (recomLine IgM IB) to 50.0% (Enzygnost IgM ELISA) (Fig. 1A). The sensitivities of the IgG assays were higher and ranged from 47.1% (IDEIA IgG) to 81.3% (recomBead IgG assay) (Fig. 1A). The overall Ig sensitivities of the antibody assays largely overlapped with those for IgG only and ranged from 47.1% (IDEIA) to 100% (Enzygnost ELISA) (Fig. 1A). The overall Ig sensitivities of two assays were slightly higher than those for IgG only, as a solitary IgM response was found for two possible LNB patients in either one (Medac ELISA) or two (Medac and Enzygnost ELISA) assays (Table S2). The sensitivity of the recomBead assay based on the overall Ig results was also slightly higher than that for IgG only, as for one possible LNB patient the IgM result was missing (Table S2). The specificities of all IgM assays were 100%, except for the Enzygnost IgM ELISA for which the specificity was 99.3% (Fig. 1A). The specificities of the IgG assays ranged from 95.7% (Serion IgG) to 100% (IDEIA IgG and Medac IgG) (Fig. 1A). The specificity of the Enzygnost ELISA based on the overall Ig results was slightly lower than that for IgG only due to a solitary IgM response for one non-LNB patient (Table S2). The sensitivity and specificity of the C6 ELISA were 94.1% and 96.4%, respectively. For IgM, IgG, and overall Ig, the sensitivity and specificity of the seven antibody assays did not differ significantly as the respective 95% confidence intervals (CIs) overlapped (Fig. 1A).

The positive predictive value (PPV) of the Enzygnost IgM ELISA was 83.3%, and the PPVs of all other IgM assays were 100%, except for the recomLine IgM IB, for which the PPV could not be calculated due to the absence of positive IgM results (Fig. 1B). For the IgG assays, the PPVs ranged from 64.3% (recomLine IgG IB) to 100% (IDEIA and Medac ELISA) (Fig. 1B). The PPVs of the antibody assays based on the overall Ig results were almost comparable to those based on IgG only (Fig. 1B). The negative predictive values (NPVs) of the IgM assays ranged from 89.1% (recomLine IgM IB) to 96.5% (Enzygnost IgM ELISA) (Fig. 1B). For the IgG assays, the NPVs ranged from 93.9% (IDEIA IgG) to 98.6% (Enzygnost IgG ELISA) (Fig. 1B). The NPVs of the antibody assays based on the overall Ig results were almost comparable to those based on IgG only (Fig. 1B). The PPV and NPV of the C6 ELISA were 76.2% and 99.3%, respectively. For IgM, IgG, and the overall Ig results, the PPVs and negative predictive values (NPVs) of the seven antibody assays did not differ significantly as the respective 95% CIs overlapped (Fig. 1B).

Random forest (RF) modeling was performed to investigate if the diagnostic performance using the antibody assays could be improved by including the results of other parameters. The seven RF models performed comparably well, which was reflected by the areas under the curve (AUCs) that ranged from 0.970 to 0.993 (Table 4). The probability of misclassification of the 156 patients ranged from 3.4% (Enzygnost ELISA RF model) to 7.1% (IDEIA and recomLine IB RF models). The sensitivities and NPVs of most RF models were higher than the upper limit of the respective 95% CIs obtained using the results of the antibody assays only, except for the C6 and the Enzygnost ELISA (Table 4 and Fig. 1A and B). In contrast, the specificities and PPVs of most RF models were comparable with those of the antibody assays only, except for the IDEIA and the Medac ELISA, for which the specificities and PPVs obtained using RF modeling were below the lower limit of the respective 95% CIs obtained using the results of the antibody assays only.

For each RF model, the relative importance of the 13 predictor variables was visualized in a heat map (Table 5). Overall, the predictor variables in each RF model ranked comparably. The most important diagnostic parameter in predicting LNB was the antibody assay with a mean rank of 1.7, followed by two-tier serology on serum (mean rank of 2.4) and CSF-CXCL13 (mean rank of 2.6). Of all Reibergram-based predictor variables, a dysfunctional blood-CSF barrier with intrathecal total antibody synthesis (i.e., Reibergram area 3) was most important with a mean rank of 4.1 and preceded pleocytosis (mean rank of 5.1). A dysfunctional blood-CSF barrier in the absence of intrathecal total antibody synthesis (i.e., Reibergram area 2) and the Borrelia species PCR on CSF contributed the least (Table 5).

Retrospectively, and regardless of their clinical presentation and age, all consecutive patients were eligible for inclusion if a CSF sample and a blood sample (drawn within 24 h of the LP) had been sent to the microbiology laboratory of the Diakonessenhuis Hospital, Utrecht, the Netherlands in the period between August 2013 and June 2016. Until the start of this study in 2017, leftover material from these patients had been stored at −20°C and/or −80°C. Prior to the start of this study, all samples had been freeze-thawed once to aliquot for use in the various assays in this study as well as for another study (45), which ran in parallel. Aliquoted samples were stored at −20°C until use. A prerequisite for patients to be included was the availability of at least 1,250 μL of CSF and 110 μL of serum. Patients from whom the CSF was (visually) hemolytic or who had received treatment with intravenous IgG were excluded from the study, as both could interfere with the test results (57, 58). One patient was excluded because of unreliable test results that implied a sample mix-up. As the final number of LNB patients among the included consecutive patients was limited, we included six patients from outside the predefined study period (from February 2011 to July 2013 and from July 2016 to November 2017) that had been diagnosed with LNB in our hospital. These additional LNB patients had taken part in two other studies of our research group for which only adult patients had been included (43, 44). Due to these previous studies, both CSF and serum from the time of diagnosis had been stored. For the CSF and serum of these additional LNB patients, the same inclusion criteria applied as for the CSF and serum of all consecutive patients.

All CSF-serum pairs used in this study were anonymized. Approval of the local ethics committee was not necessary, as the main goal of our study was to compare various antibody assays and assess if additional parameters could improve LNB diagnostics for which leftover material can be used. We did, however, obtain approval from the hospital board. The results of this study are reported following the guidelines for diagnostic accuracy studies (59).

The EFNS guidelines (3) were used to classify the patients in the current study. Following these guidelines, definite LNB patients should have (i) clinical symptoms suggestive of LNB in the absence of another cause, (ii) CSF pleocytosis (≥5 leukocytes/μL), and (iii) intrathecal synthesis of Borrelia-specific antibodies. Possible LNB patients should have clinical symptoms suggestive of LNB with either pleocytosis or intrathecally produced Borrelia-specific antibodies. Clinical symptoms suggestive of LNB (i.e., fulfillment of the first EFNS criterion) were assumed to be present when a request for the detection of intrathecal Borrelia-specific antibody synthesis was done at our laboratory at the time of active disease in the past for which the IDEIA LNB IgM and IgG assay (Oxoid, Hampshire, United Kingdom) was used. If the second and/or third criterion (i.e., pleocytosis and intrathecal Borrelia-specific antibody synthesis, respectively) was also fulfilled, another cause for the symptoms had to be excluded according to the EFNS guidelines. To minimalize the bias in patient classification by using only the IDEIA results, a consensus strategy for proof of intrathecal synthesis of Borrelia-specific antibodies was applied. This strategy entailed that intrathecal Borrelia-specific antibody synthesis was considered proven only if the majority of the antibody assays tested on CSF-serum pairs in this study showed a pathological Borrelia-specific (IgM and/or IgG) AI value (Fig. S1).

Seven commercial antibody assays were selected, which were based on different techniques, different Borrelia antigens, and different quantification methods (Table S3). For five assays, referred to here as CSF-serum assays, the detection of intrathecally produced Borrelia-specific IgM and/or IgG was done by testing the CSF and serum of each patient simultaneously. Results of these CSF-serum assays were used to calculate a Borrelia-specific AI value as described below. For two assays, Borrelia-specific IgM and/or IgG antibodies were determined in CSF only and are referred to here as CSF-only assays. The seven antibody assays were performed according to the respective manufacturer’s instructions, unless otherwise specified here.

The first CSF-serum assay was the second-generation IDEIA LNB assay (Oxoid), referred to here as IDEIA, for the detection of IgM and IgG (Table S3) (44). As the IDEIA was part of the routine LNB diagnostics in our hospital, CSF-serum pairs of patients for whom LNB was suspected had already been tested with this assay at the time of active disease in the past. Consequently, these results were retrieved from the laboratory information system. The CSF-serum pairs of patients for which a request for the detection of intrathecal Borrelia-specific antibody synthesis was not done at the time of active disease in the past were tested in batch. CSF-serum pairs were tested only in the singular due to the limited amount of sample material, except CSF-serum pairs for which the CSF sample had an optical density (OD) value of ≥0.100, which were repeated. The second CSF-serum assay was the Borrelia ELISA Medac (Medac GmbH, Hamburg, Germany), referred to here as Medac ELISA, for the detection of IgM and IgG (Table S3). The third CSF-serum assay was the recomBead Borrelia 2.0 multiplex bead-assay (Mikrogen Diagnostik GmbH, Neuried, Germany), referred to here as the recomBead assay, for the detection of IgM and IgG (Table S3). For both the Medac ELISA and the recomBead assay, CSF-serum pairs with an equivocal AI value (1.3 ≤ AI < 1.5) should have been repeated following the manufacturers’ protocol (Table S3), but this was not done due to limited sample material. The fourth CSF-serum assay tested was the Borrelia burgdorferi SERION ELISA classic (Institute Virion/Serion GmbH, Würzburg, Germany), referred to here as Serion ELISA, for the detection of IgM and IgG (Table S3). The fifth CSF-serum assay was the Enzygnost Borreliosis/IgM ELISA (Siemens Healthcare Diagnostics, Marburg, Germany) and the Enzygnost Lyme link VlsE/IgG ELISA (Siemens Healthcare Diagnostics), referred to here as the Enzygnost ELISA (Table S3). CSF-serum pairs were tested using an adapted protocol we described previously (60), as an edge effect was established for the Enzygnost IgG ELISA following the standard protocol of the manufacturer. This adapted protocol was also used for the Enzygnost IgM ELISA, for which an edge effect was established as well (data not published).

The first CSF-only assay was the C6 ELISA (Immunetics, Boston, MA, USA), which measures total Ig (IgM and IgG) (Table S3) (61). The protocol of the manufacturer describes its use only for serum. This protocol was used to test the CSF using a 1:5 dilution, similar to the CSF dilution used by van Burgel et al. (40). The second CSF-only assay was the recomLine IB (Mikrogen GmbH) for the detection of IgM and IgG (Table S3). For practical reasons, we used the manufacturer’s protocol for serum to test the CSF. For testing CSF, however, we used a 1:10 dilution for IgM and a 1:20 dilution for IgG (for serum, a 1:51 dilution for IgM and a 1:101 dilution for IgG is recommended). In January 2019, the manufacturer of the recomLine IB revised the interpretation of the recomLine IgG IB (Table S3). Both the old and revised interpretation criteria were investigated in the current study (62, 63); however, only the revised interpretation criteria were elaborated on throughout the article. For both the C6 ELISA and the recomLine IB, equivocal results were interpreted as positive.

All ELISAs were performed on a Dynex DS2 automated ELISA instrument (Dynex Technologies, Chantilly, VA, USA) and analyzed with the DS-Matrix software (Dynex Technologies). The recomBead assay was performed on a Bio-Plex 200 instrument using the Luminex xMAP technology (Bio-Rad Laboratories, Hercules, CA, USA) and the Bio-Plex Manager software version 6.1 (Bio-Rad Laboratories). The recomLine IB was performed on an Autoblot 3000 (Medtec Biolab Equipment, Hillsborough, NC, USA). Subsequently, recomLine IB strips were scanned and the intensity of the bands was recorded using recomScan software version 3.4 (Mikrogen GmbH).

To investigate the blood-CSF barrier functionality and the intrathecal total IgM and total IgG synthesis, CSF and serum concentrations of albumin, total IgM, and total IgG were determined at the start of this study by nephelometry and used to calculate the CSF/serum quotients for Q Alb, total IgM (Q IgM), and total IgG (Q IgG) as described previously (45). The Q Alb is used to assess the functionality of the blood-CSF barrier as albumin originates from the blood, and a dysfunctional blood-CSF barrier is proven if the Q Alb exceeds the age-dependent Q Alb (10). The Q IgM and Q IgG are used to assess intrathecal total antibody synthesis. If either one or both quotients show a larger increase than the Q Alb and the intrathecal fraction of total IgM and/or total IgG is larger than 10%, then intrathecal total antibody synthesis is proven (64).

For the CSF-serum pairs, either one of two calculation methods was used to calculate the Borrelia-specific IgM and IgG AI value (Table S3). For the IDEIA, intrathecal Borrelia-specific antibody synthesis was proven if the fraction of Borrelia-specific IgM (or IgG) in the CSF as part of the total amount of IgM (or IgG) in the CSF exceeded that of serum, which is expressed by a Borrelia-specific IgM (or IgG) AI value (Table S3). Following the manufacturer’s protocol, a Borrelia-specific AI value of ≥0.3 was considered pathological, and a Borrelia-specific AI value of <0.3 (or an OD value of CSF of <0.150) was considered normal. Because the IDEIA is based on a capture ELISA, a correction for a dysfunctional blood-CSF barrier was not necessary (15). For the other four CSF-serum assays, intrathecal Borrelia-specific antibody synthesis was proven by the calculation of a Borrelia-specific IgM and IgG AI value according to Reiber and Peter (10). These Borrelia-specific AI values were calculated by dividing the CSF/serum quotient of Borrelia-specific IgM (or IgG) by Q IgM (or Q IgG) by considering the blood-CSF barrier functionality. As the interpretation criteria shown in Table S3 differed slightly between the four CSF-serum assays, the cutoff for intrathecal pathogen-specific (IgM and/or IgG) antibody synthesis as described by Reiber was used (65). Thus, Borrelia-specific AI values of ≥1.5 are considered pathological, Borrelia-specific AI values between 0.6 and 1.3 are considered normal, and Borrelia-specific AI values of <0.5 are not valid. In the current study, Borrelia-specific AI values between 1.3 and 1.5 were considered normal. Furthermore, Borrelia-specific AI values were used only if these were above the assay-specific lower cutoff (Table S3).

Borrelia-specific antibodies in serum were determined previously using a two-tier protocol in which the C6 ELISA was used as a screening test and equivocal and positive C6 ELISA results were confirmed using the recomLine IB (45). The recomLine IB was also performed on C6 ELISA negative sera in order to compare the recomLine IB results obtained in CSF with those obtained in serum to gain insight into the origin of the Borrelia-specific antibodies. Similar to the recomLine IgG IB performed on CSF, for serum we also applied the old and revised interpretation criteria of the recomLine IgG IB (Table S3) (62, 63); however, only the revised interpretation criteria were elaborated on throughout the article.

Results from a number of other parameters, obtained at the time of active disease in the past, were retrieved from the patient and/or laboratory information system. These results included information about clinical symptoms, total protein and glucose concentrations in the CSF, and CSF leukocyte counts. For CSF samples with elevated erythrocyte counts (i.e., ≥1,000 erythrocytes/μL), the CSF leukocyte count was corrected by subtracting 1 leukocyte/μL for each 1,000 erythrocytes/μL according to Reiber and Peter (10). For patients classified as definite or possible LNB patient, information regarding symptom duration and antibiotic treatment for LNB was retrieved from the patient information system.

An in-house Borrelia species PCR and two CXCL13 assays on CSF had been performed previously, of which the respective methods and results have been published (45). For the current study, the final CSF-CXCL13 result was based on the combined result of the two CXCL13 assays and was considered negative when either one or both assays were negative and positive when both assays were positive.

For all assays that determined IgM and IgG separately, the overall Ig results were based on a combination of the results of both immunoglobulins: negative when both IgM and IgG were negative and positive when at least one of these was positive. For all statistical analyses, Rstudio (version 1.4.1717, 2009 to 2021) was used (66). We analyzed all data by performing two-group comparisons. The Fisher’s exact test for count data was used for unpaired nominal data, the McNemar’s chi-squared test with continuity correction was used for paired nominal data, and the exact Wilcoxon-Mann-Whitney test was used for quantitative unpaired data in a 2 by 2 table using the “stats” package (67). Unpaired nominal data in a 2 by 4 table were analyzed with the approximate Monte Carlo resampling 10⁶ Pearson’s chi-squared test using the “coin” package (68). Depending on the distribution, either the (geometric) mean value with the 95% CI or the median value and range were shown. Raw P values of <0.050 were statistically significant; however, they were interpreted after correction for the multiple statistical analyses in this study, for which the Benjamini-Hochberg procedure (BH) was applied (69). The false-discovery rate (FDR) was set at the level of 2.0% (i.e., less than one false-positive test result was allowed in our list of rejections).

For each antibody assay, the sensitivity, specificity, PPV, and NPV with 95% CIs were calculated using the “epiR” package (70), for which definite and possible LNB patients were used as cases and all non-LNB patients were used as controls.

To investigate if the diagnostic performance using the antibody assays could be improved by including the results of other parameters, RF modeling was performed to predict LNB (71). For each antibody assay, an RF model was built using the “randomForest” package (72) which included 13 predictor variables. The first six predictor variables comprised (i) one of the seven antibody assays (negative/positive), (ii) two-tier serology on serum (negative/positive), (iii) pleocytosis (no/yes), (iv) CSF-CXCL13 (negative/positive), (v) elevated total protein in CSF (no/yes), and (vi) Borrelia species PCR on CSF (negative/positive). Blood-CSF barrier functionality and intrathecal total antibody synthesis were also included in the RF models for which the previously published Reibergram classification was used (45). In short, all patients included in this study were classified to one of four of the five Reibergram areas: Reibergram area 1, a normal blood-CSF barrier without intrathecal total IgM and/or IgG synthesis of >10% (n = 107/156), Reibergram area 2, a dysfunctional blood-CSF barrier without intrathecal total IgM and/or IgG synthesis of >10% (n = 25/156), Reibergram area 3, a dysfunctional blood-CSF barrier with intrathecal total IgM and/or IgG synthesis of >10% (n = 9/156), and Reibergram area 4, intrathecal total IgM and/or IgG synthesis of >10% with a normal blood-CSF barrier (n = 15/156) (Table S2). Subsequently, seven predictor variables were constructed based on one or multiple Reibergram areas (Table 6).

For all RF models, definite and possible LNB patients were defined as cases and all non-LNB patients were defined as controls. In total, 10⁵ decision trees were built, and for each tree node, three predictor variables were used to split the tree. The predictions obtained in each RF model were used to construct a receiver operating characteristic (ROC) curve, which was subsequently used to calculate the AUC. For each RF model, the optimal cutoff for predicting LNB was calculated using the point on the ROC curve closest to the upper left corner, where both sensitivity and specificity are 100%, and this was determined by the square root of [(1 − sensitivity)² + (1 − specificity)²]. Using the optimal cutoff value, the sensitivity, specificity, PPV, and NPV of each RF model were calculated. As the performance characteristics are based on predictions, 95% CIs were not calculated. For each RF model, the probability of misclassification and the relative importance of each predictor variable were calculated as described by Liaw and Wiener (72). For each RF model, the relative importance of the 13 predictor variables was made visible by construction of a heat map. Subsequently, for each RF model, the predictor variables were ranked from 1 (highest relative contribution) to 13 (lowest relative contribution). These ranks were then used to calculate the mean rank of each predictor variable for the seven RF models to assess the importance of each predictor variable in predicting LNB.