Decision Support System Quality Assessment

Data Reliability

Data Similarity

Online tool that computes the degree of correspondence (psi), or similarity between two datasets

Calibration

Online tool to assess and visualize the calibration of a machine learning model via the Estimated Calibration Index framework (extension of the Expected Calibration Error)

Robustness

Online tool to generate two data visualizations: the Potential Robustness Diagram (PRD), which relates a set of similarities with the corresponding accuracy score (intended use to visualize internal validation scores); the External Performance Diagram (EPD), which displays the results of three performance analyses (discriminative, utility, calibration) as a function of the similarity between training/validation and external validation/test sets.

Human Interaction

Online tool to assess the quality of Human-AI Interaction (benefit and reliance) in classification decision tasks generating a set of scores and visualizations, including causal metrics which define the use of AI and/or XAI akin to pharmacological and therapeutic interventions.

DSS Quality Assessment > Calibration

Online calculator for the assessment of the quality of AI-based decision support systems, along different and complementary dimensions, such as data reliability, robustness, calibration, utility and impact on human decision making.

It's a five step journey, but you can select any tool below.

Data Reliability
Data Similarity
3
Calibration
4
Robustness
5
Human Interaction
Developed at MUDI Lab @ UniMiB