Assessing the accuracy of species distribution models: prevalence, kappa and the true skill statistic (TSS)
2006/09/12 by OMRI ALLOUCHE, Omri Allouche, ASAF TSOAR +3 · 169 citations
Environmental Science · #Species Distribution and Climate Change #Ecology and Vegetation Dynamics Studies #Wildlife Ecology and Conservation
paper · pdf · doi:10.1111/j.1365-2664.2006.01214.x
Abstract
Summary In recent years the use of species distribution models by ecologists and conservation managers has increased considerably, along with an awareness of the need to provide accuracy assessment for predictions of such models. The kappa statistic is the most widely used measure for the performance of models generating presence–absence predictions, but several studies have criticized it for being inherently dependent on prevalence, and argued that this dependency introduces statistical artefacts to estimates of predictive accuracy. This criticism has been supported recently by computer simulations showing that kappa responds to the prevalence of the modelled species in a unimodal fashion. In this paper we provide a theoretical explanation for the observed dependence of kappa on prevalence, and introduce into ecology an alternative measure of accuracy, the true skill statistic (TSS), which corrects for this dependence while still keeping all the advantages of kappa. We also compare the responses of kappa and TSS to prevalence using empirical data, by modelling distribution patterns of 128 species of woody plant in Israel. The theoretical analysis shows that kappa responds in a unimodal fashion to variation in prevalence and that the level of prevalence that maximizes kappa depends on the ratio between sensitivity (the proportion of correctly predicted presences) and specificity (the proportion of correctly predicted absences). In contrast, TSS is independent of prevalence. When the two measures of accuracy were compared using empirical data, kappa showed a unimodal response to prevalence, in agreement with the theoretical analysis. TSS showed a decreasing linear response to prevalence, a result we interpret as reflecting true ecological phenomena rather than a statistical artefact. This interpretation is supported by the fact that a similar pattern was found for the area under the ROC curve, a measure known to be independent of prevalence. Synthesis and applications . Our results provide theoretical and empirical evidence that kappa, one of the most widely used measures of model performance in ecology, has serious limitations that make it unsuitable for such applications. The alternative we suggest, TSS, compensates for the shortcomings of kappa while keeping all of its advantages. We therefore recommend the TSS as a simple and intuitive measure for the performance of species distribution models when predictions are expressed as presence–absence maps.
Citations
Cited by
- Validating habitat suitability models for pine marten ( <i>Martes martes</i> ) reintroductions to England and Wales
- Socioecological risks amplified by rising plant invasions in India
- Genetic diversity and ecological niche modelling of the restricted<i>Recordia reitzii</i>(Verbenaceae) from southern Brazilian Atlantic forest
- Projected climate change and limited dispersal potential threaten the seahorse species Hippocampus hippocampus and Hippocampus guttulatus
- Evaluating migration hypotheses for the extinct <i>Glyptotherium</i> using ecological niche modeling
- The Forest After Tomorrow: Projecting the Impact of a Collapsing Atlantic Meridional Overturning Circulation on European Tree‐Species Distributions
- Ship collision risk threatens whales across the world’s oceans
- Ecological niche models based on occurrence records of the amphibian chytrid fungus overestimate lethal chytridiomycosis risk
- Physiological and morphological traits affect contemporary range expansion and implications for species distribution modelling in an amphibian species
- 3D ecological niche models outperform 2D in predicting coelacanth (Latimeria spp.) habitat
- Factors Influencing Species Distribution Model Performance in Tropical Reef Fishes
- Restoration cannot be scaled up globally to save reefs from loss and degradation
- Assessing conservation priorities for seahorses in Brazil reveals gaps in current protected areas
- Correcting environmental sampling bias improves transferability of species distribution models
- Climate change effects on marine protected areas: Projected decline of benthic species in the North Sea
- Future climate change is predicted to shift long-term persistence zones in the cold-temperate kelp Laminaria hyperborea
- A new westward migration route in an Asian passerine bird
- Effects of climate change on the global distribution for the invasive scale <i>Ceroplastes stellifer</i> (Hemiptera: Coccidae) and implications for its management
- Prediction of potential suitable areas for 5 common <i>Coraebus</i> species (Coleoptera: Buprestidae) in China based on Biomod2 combination model
- Comparison of metrics for identifying Key Biodiversity Areas
- Potential geographic distribution of the coralsnake Micrurus decoratus Jan, 1858 (Serpentes, Elapidae) in the Atlantic Rain Forest of Brazil
- Impact of climate change on the current and future distribution of threatened species of the genus Lessingianthus (Vernonieae: Asteraceae) from the Brazilian Cerrado
- Impact of environmental barriers on temnospondyl biogeography and dispersal during the Middle–Late Triassic
- Using rare mosses to resolve barriers in the use of species distribution models for climate change vulnerability assessments
- Navigating Future Waters: The Resilience of the Atlantic Bluefin Tuna Under Climate Change
- Deciphering the enigma of human−lion coexistence in India
- RRphylogeography: A new method to find the area of origin of species and the history of past contacts between species
- Modelling the Distribution of Key Mediterranean Gorgonians: An Ensemble Approach to Unravel Broad‐Scale Patterns and Guide Conservation Efforts
- Potential effects of climate change on the threatened Malagasy poison frogs: A multispecies approach
- <scp>GLOSSA</scp> : A user‐friendly R Shiny application for Bayesian machine learning analysis of marine species distribution
- A standard protocol for reporting species distribution models
- BotaCLIP: Contrastive Learning for Botany-Aware Representation of Earth Observation Data
- Unveiling dynamic conservation gaps for highly migratory marine fishes through seasonal niche-environment alignment
- Spatial patterns of bark beetle–induced pine mortality on a Mediterranean island: Implications for forest impact assessment
- Submarine canyons and oceanographic processes shape suitable habitats of the deep-sea shrimp Aristaeopsis edwardsiana (Johnson, 1868) in the Southwestern Atlantic
- Rapid Declines in Southern Sierra Nevada Fisher Habitat Driven by Drought and Wildfire
- Forested Natura 2000 sites under climate change: effects of tree species distribution shifts
- A framework to predict zoonotic hosts under data uncertainty: A case study on betacoronaviruses
- Range Shift and Climatic Refugia for Alpine Lichens Under Climate Change
- Burrowing Into the Past: Extending Niche Space Models of Procellariiform Breeding Grounds by Merging Fossil and Historic Data
- Prevalence affects the evaluation of discrimination capacity in presence-absence species distribution models
- Using recommender systems to analyze and predict plant communities from partial assemblages
- Relative role of operational patterns and environmental conditions on cookie cutter shark damage in the Hawai‘i longline fishery
- Assessing Risk for Bycatch and Byproduct Species Using a Modified Sustainability Assessment for Fishing Effects (SAFE) Approach
- Predicting the global distribution and invasion scenarios of the Spotted Lanternfly, Lycorma delicatula (White, 1845) (Hemiptera, Fulgoridae)
- Land Cover Constrains Range Shifts in Northern Iberian Bird Species Under Climate Change Scenarios
- Predicting Road‐Crossing Passability for River Connectivity Analysis
- FINE‐SCALE HABITAT MODELING OF A TOP MARINE PREDATOR: DO PREY DATA IMPROVE PREDICTIVE CAPACITY
- How to Identify Priority Sites for Invasive Alien Species Policy and Management
- Historical human impact on the endangered, relict and iconic Canary Islands dragon tree (Dracaena draco (L.) L.) and its uncertain fate in the face of climate change
- Spatial differences in predicted <i>Phalaris arundinacea</i> (reed canarygrass) occurrence in floodplain forest understories
- The socioeconomic and environmental niche of protected areas reveals global conservation gaps and opportunities
- The impact of the invasive <i>Kalanchoe</i> × <i>houghtonii</i> on vegetated sea cliffs of the Mediterranean coasts with endemic <i>Limonium</i> species
- Decadal predictions of future habitat favourability of the European green crab (Carcinus maenas) on the Pacific Coast of North America
- Diversification and historical biogeography of the Himalayan toad (Duttaphrynus himalayanus)
- Characterizing urban landscapes using very-high resolution satellite imagery to predict Ae. albopictus larval presence probability in public spaces
- Predicting current and future distributions of bird species in South Africa’s arid Northern Cape under climate change scenarios: are biomes being crossed?
- Ecotrends: an R package for estimating habitat suitability trends over time
- Von Humboldt's Nightmare: Effects of Climate Change on the Plant Biodiversity of Mount Chimborazo (Tropical Andes)
- The role of climate change in the past, present, and future of two species of montane neotropical mice, genus Juliomys
- The elephant in the room: Comparison of Species Distribution Models for human-elephant conflict risk mapping
- Dependence of Europe's most threatened mammals on movement to adapt to climate change
- A 37 years [1984–2021] Landsat/Sentinel-2 derived snow cover time-series for Switzerland
- Targeted conservation assessment of the climate-sensitive, poorly known Pilbara barking gecko (Underwoodisaurus seorsus)
- Local niches explain coexistence in environmentally-distinct contact zones between Western Mediterranean vipers
- A genetic, demographic and habitat evaluation of an endangered ephemeral species Xerothamnella herbacea from Australia’s Brigalow belt
- The conservation of migratory fishes in the second largest river basin of South America depends on the creation of new protected areas
- Landscape‐Mediated Effects of Climate Change on Metacommunities Inhabiting Mediterranean Seagrass
- Archaeological ostrich eggshell structure as a taxonomic and paleoenvironmental indicator
- Nationwide mapping of pine mistletoe hotspots for adaptive management: MaxEnt modelling in Polish forests
- When Good Fit Goes Bad: Identifying and Minimising Overfitting in Ecological Niche Models
- Niche modelling supports cryptic diversification in an Andean hemiparasitic herb
- Species distribution modeling to predict tsetse fly (Glossina spp.) habitat suitability in Kenya
- Reshuffling of Azorean Coastal Marine Biodiversity Amid Climate Change
- Mapping the habitat suitability of <i>Culex pipiens</i> in Europe using ensemble bioclimatic modelling
- Extinction risk of the endemic soft coral Phyllogorgia dilatata (Esper, 1806) in a remote island of the Brazilian coast
- Distribution, drivers and restoration priorities of plant invasions in India
- Climate change reduces range overlap among congeneric plant species in the Iberian Peninsula and Balearic Islands
- Integrating habitat suitability, disturbance, and biotic interactions into the ecological restoration of the saguaro (<i>Carnegiea gigantea</i>) in drylands of the southwest of the United States and northern Mexico
- Saharan vertebrate flagships are good ecosystem-wide climate response surrogates
- Glacial refugia and mid-Holocene expansion delineate the current distribution of Castanea sativa in Europe
- Mapping the Habitat Suitability of <i>Culex pipiens</i> in Europe Using Ensemble Bioclimatic Modelling
- Transnational conservation to anticipate future plant shifts in Europe
- Climate change favours African malaria vector mosquitoes
- Refined gap analysis for biodiversity conservation under climate change
- Unprecedented distribution data for Joshua trees (Yucca brevifolia and Y. jaegeriana) reveal contemporary climate associations of a Mojave Desert icon
- Forecasting the future of a desert relict: current and future potential distribution of the rare Argun palm Medemia argun under climate change scenarios
- Predicting range shifts of African apes under global change scenarios
- Climate change impacts the distribution and suitability of two wild species of the genus Stylosanthes
- Climate Change Impacts on Suitable Habitats of the Endangered Parnassius imperator, an Alpine Butterfly Endemic to China
- Without quality presence–absence data, discrimination metrics such as <scp>TSS</scp> can be misleading measures of model performance
- Identifying life-history patterns along the fast-slow continuum of mammalian viral carriers
- Nowhere to hide: Ensemble of Small Models exposes climate vulnerability and conservation gaps for an extremely rare troglobitic beetle genus
- Understanding habitat selection of range‐expanding populations of large carnivores: 20 years of grey wolves ( <i>Canis lupus</i> ) recolonizing Germany
- Assessment of the vulnerability of Spiny Lizards in Central Mexico under anthropogenic pressure: integrating habitat suitability, landscape connectivity, fragmentation and protected area coverage
- Environmental specialization decouples geographic range and habitat occupancy in Ragala ucuquirana-branca (Sapotaceae) across the Amazon Basin
- The current distribution of an invasive moss species Campylopus introflexus (Leucobryaceae) and models of its future range dynamics in two contrasting climatic scenarios in Europe
- Bat Migration Intensifies Cave Fish Richness Loss Under Climate Change in China
- Ground‐Truthing of <scp>MaxEnt</scp> Models Reveals Poor Predictive Accuracy for Lizards in the Mackenzie Basin, New Zealand
- Designing a Climate Change Resilient Landscape Connectivity Network From a Multi‐Species Perspective
- Measuring and comparing the accuracy of species distribution models with presence–absence data
- Developing and Validating Species Distribution Models for Wetland Plants Across Europe
- Multiple Threshold‐Selection Methods Are Needed to Binarise Species Distribution Model Predictions
- Combining Hierarchical Distribution Models With Dispersal Simulations to Predict the Spread of Invasive Plant Species
- Turning the Tide: A 2°C Increase in Heat Tolerance Can Halve Climate Change‐Induced Losses in Four Cold‐Adapted Kelp Species
- Morphological traits distinguish feeding guilds in a Southern Ocean demersal fish community
- Climate and Dispersal Ability Limit Future Habitats for Gila Monsters in the Mojave Desert
- Predictive Modeling of Juvenile Smalltooth Sawfish Habitats: Challenges and Opportunities for Conservation
- Ensemble Niche Modelling Projects Net Suitability Gain and Eastward Range Expansion for the Namaqua Dove (Oena capensis) in Anatolia Under Climate Change
- Comparative modelling of two migratory locusts along the China–Kazakhstan border under climate change: Poleward habitat shifts and increasing transboundary risk
- From research to conservation: Site selection for habitat restoration of a narrowly distributed and critically endangered butterfly
- Revealing hidden patterns in the potential distribution of Leposternon polystegum (Duméril, 1851) (Amphisbaenia, Amphisbaenidae) using fuzzy logic modeling
- Biogeography of pollination under climate change: integrating pollinator effectiveness into species distribution models for mango in South Africa
- Interactions between intrinsic and extrinsic factors shape island bat survival and extirpation in a global extinction hotspot
- The invasive woodwasp Sirex noctilio Fabricius threatens pine forest carbon storage in China under climate change
- Comprehensive spatial risk assessment using community-based species distribution models stratified by ant damage types
- Climate change reshapes species-specific connectivity among marine protected areas in the Yangtze River estuary and adjacent waters
- Modeling the potential distribution of Musa (AAB), cv. ‘Bellaco’ and Carica papaya under climate change scenarios in the provinces of Jaén and San Ignacio
- Habitat suitability and range mapping of the Philippine endemic Tectona philippinensis for spatial conservation
- Habitat Suitability Assessment of Milu (Elaphurus davidianus) in Coastal Wetlands of Jiangsu Province Based on Species Distribution Models
- Conservation priorities for Neotropical water striders (Hemiptera: Heteroptera: Gerridae) under climate change
- Predicting habitat changes for cold-water freshwater fish under climate change scenarios using a machine-learning-based ensemble species distribution model
- Machine learning-based ensemble species distribution models to guide monitoring and survey design for offshore wind
- Climate refugia in a warming ocean: salinity and temperature shape the persistent core habitat of Sthenoteuthis oualaniensis in the Indian Ocean
- Spatial distribution pattern of Pavetta crassipes K. Schum. under global change and its implications for conserving savanna ecosystems in West Africa
- Multi-scale modeling of habitat suitability and human-induced risk for the critically endangered white-rumped vulture (Gyps bengalensis) in South and Southeast Asia
- Habitat suitability modeling reveals conservation gaps for endemic canga flora in Brazil’s Iron Quadrangle
- Global habitat suitability and invasion risk of melon thrips
- Climatic Niche Contraction and Refugial Persistence of an Invasive Tephritid Pest Across the Arabian Peninsula Under Contrasting Emission Scenarios
- Integrated spatiotemporal dynamics and spatial prediction of medicinal quality of Aconitum tanguticum based on species distribution models and machine learning approaches
- Sensitivity of habitat suitability-derived connectivity models to three-dimensional measures of urban landscape structure
- A framework for climate-resilient forest planning and restoration: advances linking species distribution modelling, genetic adaptation and future climate scenarios
- Identifying suitable habitats under climate change for non-targeted demersal fish in the Mediterranean Sea
- Projected habitat loss and persistence of Heortia vitessoides (Lepidoptera, Crambidae) in China Under CMIP6 Climate Models
- Habitat Dynamics and Protected Area Effectiveness of the Endangered <i>Paphiopedilum</i> Subgen. <i>Brachypetalum</i> (Orchidaceae) Under Climate Change
- Predicting the Current and Future Distribution of the Invasive Weed Ageratina adenophora in the Chitwan–Annapurna Landscape, Nepal
- Modeling the potential global distribution of the invasive Jack Beardsley mealybug (Hemiptera: Pseudococcidae) under climate change
- Hybrid Projections Improve Prediction of Distributional Shifts of Invasive and Native Seaweeds Under Climate Change
- Proximal microclimate: Moving beyond spatiotemporal resolution improves ecological predictions
- Phylogeography and ecological niche modelling reveal a geographical break in the genetic structure of <i>Boana crepitans</i> (Anura: Hylidae) during the Pleistocene in Eastern Brazil
- Evaluating climate change impacts on habitat suitability for Iberian freshwater crab “Potamon ibericum Bieberstein, 1808” in Iran
- Climate-driven redistribution of global Haloxylon Bunge habitat revealed by ensemble species distribution models: Implications for dryland restoration indicators
- Population, expanded distribution, and conservation of the Sumatran endemic tree Magnolia sumatrae (Dandy) Figlar &amp; Noot.
- Identification of potential suitable habitats and high conservation value areas for threatened species in Rajah Sikatuna Protected Landscape, Philippines
- Spatial analysis improves species distribution modelling during range expansion. [europepmc]
- Genetic consequences of climate change for northern plants. [europepmc]
- Ecological Niche Modelling Predicts Southward Expansion of Lutzomyia (Nyssomyia) flaviscutellata (Diptera: Psychodidae: Phlebotominae), Vector of Leishmania (Leishmania) amazonensis in South America, under Climate Change. [europepmc]
- Why choose Random Forest to predict rare species distribution with few samples in large undersampled areas? Three Asian crane species models provide supporting evidence. [europepmc]
- Global hotspots and correlates of emerging zoonotic diseases. [europepmc]
- Habitat suitability and movement corridors of grey wolf (Canis lupus) in Northern Pakistan. [europepmc]
- Insights from modeling studies on how climate change affects invasive alien species geography. [europepmc]
- Ecological niche modeling predicting the potential distribution of Leishmania vectors in the Mediterranean basin: impact of climate change. [europepmc]
- Evaluating collinearity effects on species distribution models: An approach based on virtual species simulation. [europepmc]
- Climate change-driven range losses among bumblebee species are poised to accelerate. [europepmc]
- Changes in human footprint drive changes in species extinction risk. [europepmc]
- Uncertainty in ensembles of global biodiversity scenarios. [europepmc]
- Xylella fastidiosa: climate suitability of European continent. [europepmc]
- Extinction debts and colonization credits of non-forest plants in the European Alps. [europepmc]
- Why Cohen's Kappa should be avoided as performance measure in classification. [europepmc]
- Modeling current and future global distribution of Chrysomya bezziana under changing climate. [europepmc]
- Niche shifts and environmental non-equilibrium undermine the usefulness of ecological niche models for invasion risk assessments. [europepmc]
- Current and Projected Distributions of Aedes aegypti and Ae. albopictus in Canada and the U.S. [europepmc]
- Update of the Scientific Opinion on the risks to plant health posed by Xylella fastidiosa in the EU territory. [europepmc]
- Global distribution and conservation status of ecologically rare mammal and bird species. [europepmc]
- SDMtune : An R package to tune and evaluate species distribution models. [europepmc]
- Genomic mechanisms of climate adaptation in polyploid bioenergy switchgrass. [europepmc]
- Trends and gaps in the use of citizen science derived data as input for species distribution models: A quantitative review. [europepmc]
- Global potential distribution prediction of Xanthium italicum based on Maxent model. [europepmc]
- Global Protected Areas as refuges for amphibians and reptiles under climate change. [europepmc]
Related