INTRODUCTION
Leptospirosis is an important zoonotic disease worldwide, with its frequency and severity increasingly recognized.1,2 It has also been shown to be a significant cause of meningoencephalitis in Laos and Thailand.3 Leptospirosis is caused by Leptospira spp. spirochetes contracted by humans through exposure to environments contaminated by urine of infected mammals.2 It is estimated that ∼853,000 people are infected and 48,000 die annually.4 Most of the cases occur in the tropics, particularly in urban slums and rural areas where people are exposed to contaminated water.2 The clinical presentation of leptospirosis is often nonspecific, and as the organism does not grow well in conventional blood cultures, diagnosis is difficult, requiring sophisticated serological and molecular tests. However, vast areas of the tropics where leptospirosis is endemic have extremely limited diagnostic laboratory capacity.5 Even where the laboratory capacity exists, diagnosis using specific culture or serological microscopic agglutination test (MAT) methods2 requires considerable expertise that is not widely available, and results are only available weeks after the initial clinical presentation. At this point, no clear guidance by international bodies such as the World Health Organization (WHO) exists as to which test is recommended for acute detection. Conventionally, the observation of a 4-fold rise between the acute and convalescent sample is considered a clear indication of an acute infection and is therefore considered the gold standard; however, a recent modeling analysis has highlighted the pitfalls of this approach.6 Several manufacturers have developed rapid diagnostic tests (RDTs) for use at the bedside or point-of-care7 of which so far, none has been approved by a stringent regulatory authority. The simplicity and relatively low cost of these tests make them potentially well suited for use in resource-poor settings with limited laboratory and human capacity, as has been achieved with malaria RDTs. Evaluations of RDTs detecting IgM against Leptospira spp. antigens have been conducted, and their diagnostic characteristics have been reported to vary between areas of low and high endemicity.8 Goris et al.8 reported 69% sensitivity and 96% specificity for the LeptoTek lateral flow test when used on admission sera in a Dutch population, whereas the same test used in a Southeast Asian hospital setting (Lao PDR) had only 45% sensitivity and 75% specificity.9 These differences are very important, as a test may be well suited to one setting but not to another. It is likely that the differences, particularly for specificity, are mainly due to background antibody levels in patients who have had multiple exposures to the pathogen, similar to the challenges faced with Orientia tsutsugamushi (scrub typhus) diagnosis in endemic areas.10
To understand these challenges and identify an RDT that is suitable for use in an endemic setting for populations repeatedly exposed to the pathogen, on-site evaluations are necessary. Our study aimed to compare the diagnostic characteristics of four RDTs for leptospirosis to guide local and regional health authorities in their search for a suitable diagnostic tool to incorporate into rapid diagnostic panels in the region.
MATERIALS AND METHODS
Study population.
Consecutive patients were enrolled in the 2014 rainy season in Mahosot Hospital (longitude 17°96′04·4″N, latitude 102°61′19·1″E) in Vientiane, Lao PDR (Laos), as part of an ongoing febrile illness study.11 Patients admitted to any ward with fever < 1 month (either history of fever during this illness or documented fever > 38.0°C by axillary temperature) plus at least one of the following symptoms (indicative of leptospirosis or typhus): headache, rash, eschar, myalgia, arthralgia, lymphadenopathy, meningitis, encephalitis, respiratory symptoms (cough, crepitations, respiratory rate > 20/minutes), clinical jaundice, or acute renal failure (creatinine > 120 μmol/L) were eligible.
Ethics statement.
Study patients provided written informed consent. In case of children, a parent or guardian provided informed consent on their behalf. Ethical approval for all investigations was granted by the Oxford Tropical Research Ethics Committee, University of Oxford, United Kingdom, and the National Ethics Committee for Health Research, Laos. All samples were anonymized using a unique identifier in all procedures and analysis.
Testing procedure.
All RDTs were performed on fresh serum within 24 hours of receipt in the Mahosot Hospital Microbiology Laboratory. Sera were refrigerated before analysis. The same three laboratory technicians performed all tests for all patients, blinded to each other’s results. Four RDTs, all detecting Leptospira IgM, were compared using only the admission sample: “Test-it” (Life Assay, Product Code: LEPTO01, South Africa; N = 418), “Leptorapide” (Linnodee, owslips.com/linnodee/ordering.html, Northern Ireland; N = 492), “Dual Path Platform” (DPP) (Chembio, Medford, NY, not commercially available; N = 530), and “SD-IgM” (Standard Diagnostics/Alere, not available at the time of writing, South Korea; N = 481). Serum (5 or 10 μL, as appropriate) was used for all assays and the tests were performed according to the manufacturers’ instructions. Different numbers are because of varying numbers of donated tests and other logistical considerations (e.g., start of test inclusion due to test availability at Mahosot Hospital).
Reference diagnostics.
Leptospiral MATs were performed and interpreted by the WHO Collaborating Center for Reference and Research on Leptospirosis, Australia. A 4-fold increase between admission and convalescent samples was considered “evidence of acute infection,” whereas a 2-fold increase/decrease or an admission titer ≥ 1:400 was considered “evidence of recent infection.” Only a subset of patients had both admission and follow-up sample available for MAT testing (N = 248) and therefore samples positive only at admission as well as a 4-fold rise were combined for accuracy analysis. Quantitative real-time PCR (qPCR) was performed on 687/695 (98.8%) patients using DNA extracted from serum (sample taken at presentation) to detect the rrs gene according to previously described protocols.12 Cultures were performed on blood clots, remaining after removal of serum, as described previously.3
Analysis.
Data were analyzed using STATA 10.0 (Stata Corp., College Station, TX) and MedCalc for Windows, version 15.0 (MedCalc Software, Ostend, Belgium). Direct (qPCR, culture) and indirect (MAT, acute, and recent infection) diagnostic tests were combined to give a composite13 reference standard (unless stated otherwise) which was used to calculate the diagnostic accuracy values (sensitivity, specificity, and positive and negative predictive values [PPV and NPV]). To assess the interobserver agreement, kappa was calculated between the three readers for the subset of tests that had been read by all readers (1–3). Diagnostic performance characteristics (sensitivity/specificity/NPV/PPV) for the different tests were based on results obtained from only Reader 1 as all tests had been read by this reader, whereas only a subset was read by Reader 2 and 3. STARD checklist attached in supplement ST1.
RESULTS
Patient characteristics.
Between May 2014 and January 2015, 728 consecutive patients met the inclusion criteria (728/1,324, 55.0%). Of these, 33 were excluded from analysis, as insufficient sample for any of the reference testing (MAT and/or PCR) was available. Hence, the final number of patients included in the analysis was 695 (Figure 1). The majority were men (407/695, 58.6%) with a median age of 39 years (range: 0.5–92). Patients presented with a median of 5 days of fever (interquartile range: 3–7). Of all patients included, 39/695 (5.6%) were positive using the composite reference standard (MAT/qPCR/culture); 12/695 (1.7%) were positive for leptospirosis by qPCR alone and 27/695 (3.9%) by MAT (“evidence of acute infection”: 10/248; “evidence of recent infection”: 37/695). No patient was only positive by culture (n[culture] = 4). The overall positivity rate of the different RDTs ranged from 5.80% to 38.1% (“Test-it”: 154/418, 36.8%; “SD-IgM”: 28/481, 5.8%; “DPP”: 202/530, 38.1%; and “Leptorapide”: 117/492, 23.8%) in the tested subset.
Flow of participants, reference, and investigated tests. * Excluded because of lack of sample for reference testing.
Citation: The American Journal of Tropical Medicine and Hygiene 98, 4; 10.4269/ajtmh.17-0702
Diagnostic performance characteristics.
The diagnostic accuracy and respective confidence intervals (CIs) of the RDTs ranged considerably between the different tests when using the composite gold standard (Table 1, Figure 2). For Reader 1, sensitivity ranged from 17.9% to 63.6% between RDTs, with the “Test-it” assay displaying the highest diagnostic sensitivity. Sensitivities ranged from 37.5% to 66.7% when comparing RDT results with the small subset of patient with “evidence of acute infections” (Test-it: 4/6, 66.7%; SD: 3/8, 37.5%; DPP: 6/9, 66.7%; and Leptorapide: 5/8, 62.5%). When using the composite gold standard, the range of specificities was also wide (62–97%) for all investigated RDTs, with the SD-IgM assay displaying the highest diagnostic specificity (Figure 2).
Diagnostic characteristics by test and reader using a composite reference standard (MAT/culture/qPCR) including the agreement between readers estimated using kappa
| Assay | Parameter | Reader 1* (95% CI) | Reader 2† (95% CI) | Reader 3‡ (95% CI) | Kappa |
|---|---|---|---|---|---|
| Test-it | Sensitivity | 71.0% (41.9–91.6) | 62.5% (24.5–91.5) | 80.0% (44.4–97.5) | 0.56 |
| Specificity | 64.6% (59.8–69.3) | 69.5% (63.2–75.4) | 48.5% (41.3–55.7) | ||
| SD IgM | Sensitivity | 21.1% (6.1–45.6) | 12.5% (0.3–52.7) | 26.7% (7.8–45.4) | 0.73 |
| Specificity | 94.8% (92.6–96.7) | 95.7% (92.3–97.9) | 93.9% (90.4–96.8) | ||
| DPP | Sensitivity | 35.0% (15.4–59.2) | 60.0% (14.7–94.7) | 42.1% (20.3–66.5) | 0.81 |
| Specificity | 62.1% (57.7–66.4) | 54.5% (44.2–64.4) | 58.7% (54.5–63.9) | ||
| Leptorapide | Sensitivity | 47.4% (24.5–71.1) | 25.0% (3.2–65.1) | 53.3% (26.6–78.7) | 0.96 |
| Specificity | 77.2% (73.1–80.9) | 85.8% (80.7–90.1) | 66.9% (60.8–72.7) |
CI = confidence intervals; DPP = dual path platform; MAT = microscopic agglutination test. Reader 1 read all tests for all patients included in the study. Reader 2 and 3 read a subset of the tests in varying order after Reader 1. Kappa was calculated on the subset that was read by all readers (“Test-it”: 90, “SD IgM”: 63, “DPP”: 78, and “Leptorapide”: 63).
Reader 1: “Test-it”: 418, “SD-IgM”: 480, “DPP”: 530, and “Leptorapide”: 492.
Reader 2: “Test-it”: 242, “SD-IgM”: 332, “DPP”: 106, and “Leptorapide”: 242.
Reader 3: “Test-it”: 206, “SD-IgM”: 411, “DPP”: 474, and “Leptorapide”: 272.
Diagnostic characteristics for patients with ≤ 5 or > 5 days of fever before presentation. Rapid diagnostic test (RDT) results are compared with a composite reference standard (MAT/culture/qPCR) according to fever duration. (A) Sensitivity (95% CI), specificity (95% CI), PPV (95% CI), and NPV (95% CI) are shown for all RDTs with a subset of patients who presented with five or less days of fever (Test-it: N = 223; SD-IgM: N = 255; DPP: N = 289; and Leptorapide: N = 260). (B) Sensitivity (95% CI), specificity (95% CI), PPV (95% CI), and NPV (95% CI) are shown for all RDTs with a subset of patients who presented with more than 5 days of fever (Test-it: N = 191; SD-IgM: N = 217; DPP: N = 229; and Leptorapide: N = 226). All presented results are based on Reader 1. Dotted lines are included to highlight 50%, 80%, and 95%. CI = confidence intervals; DPP = dual path platform; MAT = microscopic agglutination test; NPV = negative predictive value; PPV = positive predictive value.
Citation: The American Journal of Tropical Medicine and Hygiene 98, 4; 10.4269/ajtmh.17-0702
None of the tests performed with a sensitivity and specificity of > 80% for detecting leptospirosis in admission samples, when comparing the tests with the composite reference standard. The “Test-it” RDT performed better in patients reporting 5 days of illness or less, whereas sensitivity dropped in patients presenting later (Figure 2). The DPP and SD assays performed better for patients reporting more than 5 days of illness, but all tests had large 95% CIs because of small number of positives (Figure 2). The performance of the Leptorapide test was similar at any day of presentation with sensitivities never reaching 50%. None of the tests showed significantly better sensitivity performance, with all CIs overlapping. In contrast, the SD-IgM test showed significantly better specificity in all patients, regardless of days of illness. PPVs for all investigated tests were very low with many false positives regardless of the manufacturer, test comparator, or reported days of illness (Figure 2).
Interobserver variability.
For the subset of tests that were read by all readers, the diagnostic accuracy values varied greatly, indicating that readers interpreted results differently (Table 1). Sensitivity ranged by 10–30% depending on the assay, when different laboratory technicians read a subset of the results. The least concordance between readers was recorded for the lateral flow–based “Test-it” assay (kappa: 0.56), whereas the agglutination-based “Leptorapide” assay (kappa: 0.96) was most consistently interpreted by the three readers. Of the three lateral flow–based tests, the DPP had the highest agreement (kappa: 0.81).
DISCUSSION
Given the global environmental presence of Leptospira spp. and that they have been identified as an important cause of fever in many large non-malarial fever studies,14–16 a simple, rapid diagnostic tool for diagnosing leptospirosis could have a large impact on patient care globally. In this study, we evaluated four RDTs which all detect anti-Leptospira IgM. The “Test-it” and “SD-IgM” are designed as simple lateral flow tests, whereas the “Leptorapide” is an agglutination test and the “DPP” is a lateral flow test with a unique dual path (DPP) technology.8,17 Although the three cassette-based tests represent familiar, supposedly simple-to-interpret, platforms, there was considerable interobserver variability between the three readers in this study. This was less the case for the Leptorapide test, which is an agglutination test. It is conceivable that in some cases, a delay in reading results may have occurred between the three readers that could have contributed to the observed inter-reader variability due to fading/intensifying of bands over time. Although this observation might not be representative because of the very small sample size, it is important to follow our findings up with more research to support product improvement efforts. When using a composite reference standard as comparators for the diagnostic accuracy assessment, no clearly superior RDT could be identified. The DPP assay performed consistently regardless of the days of illness with a sensitivity between 50% and 60% and specificity around 70%, which is in line with what was previously published for mild leptospirosis cases at admission as well as healthy slum habitants.18 In comparison to previously published sensitivity and specificity of more than 90%,17 we found that the “Leptorapide” assay showed a lower sensitivity (< 50%) and specificity (∼80%) combined with an NPV of ∼95%. The “Test-it” assay had a high sensitivity of ∼80% in patients with less than 5 days of fever and the specificity of the test was low at ∼70%. Earlier evaluations3,6 of this assay reported a higher specificity, and the difference can likely be explained by the fact that our study population consisted of individuals who had multiple episodes of exposure to Leptospira spp. It must be noted that one additional reason for the different results in different studies for all the tests could also be due to batch variations related to substandard manufacturing.
One significant limitation of our study is that not all tests were performed on all samples because of logistical challenges. This might have influenced the comparability of results between tests. In addition, no extensive comparison was drawn to severity of infections as the study aim was to understand the diagnostic usability to identify leptospirosis in the general population before progressing to severe disease.
The very low sensitivity of the SD-IgM assay makes it unsuitable for use as single diagnostic test in Laos, unless combined with a secondary test. It could be envisioned that combining a high sensitive, but low specificity test with a low sensitivity but high specificity test to provide more accurate diagnosis to patients. A similar “screen-and-confirm” approach is taken with human immunodeficiency virus or Hepatitis C where positive high-sensitive screening tests are followed up with more specific confirmatory tests.19,20 One could hypothesize that screening with the “Test it” or DPP test and retesting all positives with the SD-IgM assay could provide more confidence in the diagnosis where molecular testing or MAT is not possible.
The data form this study confirm that local validations are important to understand the performance of a test in a population with particular health-seeking behavior or disease epidemiology.21 Furthermore, to allow wider decentralization of tests in the future, the expansion of sample types to whole blood would very much improve the usability of the test beyond central facilities. In conclusion, none of the tests evaluated in this study showed both sensitivity and specificity > 90%, which is disappointing but not surprising, given previous reports from endemic areas, including Laos.9 This is particularly important for diagnostic assays that detect the host–antibody response rather than directly detecting the pathogen, underlining the need for combined antigen/antibody detection or other improvements in the testing algorithm such as screen and confirm, where possible.
Acknowledgments:
We are very grateful to the patients and to Associate Professor Bounthaphany Bounxouei, Director of Mahosot Hospital, the staff of the Microbiology Laboratory, LOMWRU and wards, Assistant Professor Chanphomma Vongsamphan, Director of Department of Health Care, Ministry of Health, and Professor Bounkong Syhavong, Minister of Health, Laos, for their help and support. We would like to express our gratitude to Chembio, LifeAssay, and Linodee who donated test kits to this investigation, but otherwise had no input to the design and analysis of the study. Furthermore, we thank Prof Albert Ko (Yale School of Public Health) for discussions and support when setting up the study.
REFERENCES
- 1.↑
Forbes AE, Zochowski WJ, Dubrey SW, Sivaprakasam V, 2012. Leptospirosis and Weil’s disease in the UK. QJM 105: 1151–1162.
- 3.↑
Dittrich S et al. 2015. Orientia, Rickettsia, and Leptospira pathogens as causes of CNS infections in Laos: a prospective study. Lancet Glob Health 3: e104–e112.
- 4.↑
Costa F, Hagan JE, Calcagno J, Kane M, Torgerson P, Martinez-Silveira MS, Stein C, Abela-Ridder B, Ko AI, 2015. Global morbidity and mortality of leptospirosis: a systematic review. PLoS Negl Trop Dis 9: e0003898.
- 5.↑
Petti CA, Polage CR, Quinn TC, Ronald AR, Sande MA, 2006. Laboratory medicine in Africa: a barrier to effective health care. Clin Infect Dis 42: 377–382.
- 6.↑
Limmathurotsakul D, Turner EL, Wuthiekanun V, Thaipadungpanit J, Suputtamongkol Y, Chierakul W, Smythe LD, Day NP, Cooper B, Peacock SJ, 2012. Fool’s gold: why imperfect reference tests are undermining the evaluation of novel diagnostics: a reevaluation of 5 diagnostic tests for leptospirosis. Clin Infect Dis 55: 322–331.
- 7.↑
Picardeau M, Bertherat E, Jancloes M, Skouloudis AN, Durski K, Hartskeerl RA, 2014. Rapid tests for diagnosis of leptospirosis: current tools and emerging technologies. Diagn Microbiol Infect Dis 78: 1–8.
- 8.↑
Goris MG, Leeflang MM, Loden M, Wagenaar JF, Klatser PR, Hartskeerl RA, Boer KR, 2013. Prospective evaluation of three rapid diagnostic tests for diagnosis of human leptospirosis. PLoS Negl Trop Dis 7: e2290.
- 9.↑
Blacksell SD et al. 2006. Limited diagnostic capacities of two commercial assays for the detection of Leptospira immunoglobulin M antibodies in Laos. Clin Vaccine Immunol 13: 1166–1169.
- 10.↑
Blacksell SD, Bryant NJ, Paris DH, Doust JA, Sakoda Y, Day NP, 2007. Scrub typhus serologic testing with the indirect immunofluorescence method as a diagnostic gold standard: a lack of consensus leads to a lot of confusion. Clin Infect Dis 44: 391–401.
- 11.↑
Phetsouvanh R et al. 2006. Causes of community-acquired bacteremia and patterns of antimicrobial resistance in Vientiane, Laos. Am J Trop Med Hyg 75: 978–985.
- 12.↑
Thaipadunpanit J et al. 2011. Diagnostic accuracy of real-time PCR assays targeting 16S rRNA and lipL32 genes for human leptospirosis in Thailand: a case-control study. PLoS One 6: e16236.
- 13.↑
Naaktgeboren CA, Bertens LC, van Smeden M, de Groot JA, Moons KG, Reitsma JB, 2013. Value of composite reference standards in diagnostic research. BMJ 347: f5605.
- 14.↑
D’Acremont V, Kilowoko M, Kyungu E, Philipina S, Sangu W, Kahama-Maro J, Lengeler C, Cherpillod P, Kaiser L, Genton B, 2014. Beyond malaria—causes of fever in outpatient Tanzanian children. N Engl J Med 370: 809–817.
- 15.
Mayxay M et al. 2013. Causes of non-malarial fever in Laos: a prospective study. Lancet Glob Health 1: e46–e54.
- 16.↑
Mueller TC et al. 2014. Acute undifferentiated febrile illness in rural Cambodia: a 3-year prospective observational study. PLoS One 9: e95868.
- 17.↑
Brownlow T, Kavanagh OV, Logan EF, Hartskeerl RA, Savage R, Palmer MF, Krahl M, Mackie DP, Ellis WA, 2014. ‘Leptorapide’ - a one-step assay for rapid diagnosis of human leptospirosis. Epidemiol Infect 142: 1182–1187.
- 18.↑
Nabity SA et al. 2012. Accuracy of a dual path platform (DPP) assay for the rapid point-of-care diagnosis of human leptospirosis. PLoS Negl Trop Dis 6: e1878.
- 19.↑
(CDC) CoDC. HIV Testing Guidelines. Available at: https://www.cdc.gov/hiv/pdf/guidelines_testing_recommendedlabtestingalgorithm.pdf. Accessed August 23, 2017.
- 20.↑
Firdaus R, Saha K, Biswas A, Sadhukhan PC, 2015. Current molecular methods for the detection of hepatitis C virus in high risk group population: a systematic review. World J Virol 4: 25–32.
- 21.↑
Chappuis F, Alirol E, d’Acremont V, Bottieau E, Yansouni CP, 2013. Rapid diagnostic tests for non-malarial febrile illness in the tropics. Clin Microbiol Infect 19: 422–431.