Objective
The objective of this project is to develop robust evaluation benchmarks for assessing the quality of sensor data, with a particular focus on evaluating imputed values in healthcare time-series datasets. The project investigates the limitations of conventional predictive accuracy metrics and proposes comprehensive evaluation frameworks that combine multiple complementary metrics to provide a more reliable assessment of data quality.
Description
High-quality data is fundamental to trustworthy artificial intelligence and machine learning systems. In healthcare and other sensor-driven applications, missing values are commonly reconstructed using statistical and machine learning-based imputation techniques. However, determining whether an imputation method has produced meaningful and reliable reconstructions remains an open research challenge.
Current state-of-the-art studies predominantly evaluate imputation quality using predictive accuracy metrics such as Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), and Mean Absolute Percentage Error (MAPE). While these metrics quantify the numerical difference between original and reconstructed values, they primarily assess prediction accuracy and often fail to capture important characteristics of physiological time-series data, including temporal dynamics, distributional similarity, and preservation of clinically relevant patterns.
Our research demonstrates that relying on a single evaluation metric can lead to misleading conclusions. Different metrics frequently rank imputation methods differently, indicating that no individual measure provides a complete assessment of reconstruction quality. Therefore, evaluating imputed data requires a multidimensional perspective that considers both point-wise reconstruction accuracy and preservation of the underlying statistical properties of the original signal.
This project develops comprehensive evaluation benchmarks that integrate predictive accuracy metrics with statistical distance measures and interpretable machine learning techniques. The goal is to establish standardized, application-aware evaluation methodologies that better reflect the true quality of reconstructed sensor data and improve the selection of imputation techniques for downstream machine learning tasks.
Current Research
The current research investigates the strengths and limitations of existing evaluation metrics for assessing imputation quality in physiological sensor data. Our studies demonstrate that commonly used metrics such as RMSE, MAE, and MAPE often provide inconsistent rankings of imputation methods because they measure only numerical proximity between original and reconstructed values. To overcome these limitations, we also evaluate statistical distance metrics, including Jensen-Shannon Distance (JSD) and Cohen’s Distance Test (CDT), which measure how well the reconstructed data preserves the statistical characteristics of the original signal.
Building upon these findings, we proposed FRAM-SHAP (Framework for Combined Evaluation Metrics through SHAP Analysis), a novel framework that combines multiple predictive and statistical evaluation metrics into a single weighted benchmark. Instead of assigning equal importance to every metric, FRAM-SHAP employs XGBoost, Optuna optimization, and SHAP analysis to determine data-driven weights based on each metric’s contribution to evaluating imputation quality. This provides a flexible and interpretable evaluation framework that can be adapted to different datasets, applications, and quality assessment objectives.
Future Direction
Future work will extend the proposed evaluation benchmark by incorporating additional evaluation metrics that capture temporal consistency, signal morphology, clinical relevance, and downstream machine learning performance. The framework will also be expanded beyond heart rate signals to other physiological and multimodal sensor data, including glucose, accelerometer, and stress monitoring datasets.
Furthermore, we aim to develop a generalized evaluation benchmark that automatically selects and weights evaluation metrics according to the characteristics of the dataset, the imputation technique, and the intended downstream application. By integrating explainable artificial intelligence methods with domain-specific knowledge, the project seeks to establish a standardized and adaptable framework for evaluating data quality across a wide range of healthcare and sensor-based AI applications.
Related Publications
- Gupta, V., Maleshkova M. ‘Beyond Accuracy: Assessment of Statistical Imputation techniques for Heart Rate Data’, BIO Web of Conferences, vol. 195, Article No. 03002, pp. 1–10, 2025. DOI: https://doi.org/10.1051/bioconf/202519503002
- Gupta, V., Grensing, F., van den Boom, L., and Maleshkova, M., ‘FRAM-SHAP: Framework for Combined Evaluation Metrics through SHAP Analysis,’ Proceedings of the 2025 IEEE 25th BIBE, Athens, Greece, pp. 444–448, 2025. DOI: https://doi.org/10.1109/BIBE66822.2025.00080
Letzte Änderung: 6. August 2026