Dataset

The dataset used for the study consists of 17,114 carefully curated images, specifically designed to facilitate the advancement of the field of dermatology and computational diagnostics in melanoma detection [29]. It combines multiple International Skin Imaging Collaboration (ISIC) datasets from 2018, 2019, and 2020 to create a balanced and reliable resource for training and evaluation [30]. A total of 1,249 images were collected from the 2018 dataset (174 malignant, 1,075 benign), 5,660 from the 2019 dataset (4,439 malignant, 1,221 benign), and 10,205 from the 2020 dataset (582 malignant, 9,623 benign). These images were taken directly from the official ISIC website and were selected to minimize data imbalance and enhance the representativeness of both malignant and benign cases. Although the images vary in size, they were resized during training to suit the requirements of each architecture. Although additional examples from the ISIC datasets could have been included, this would have increased the data imbalance.

The images are available in both DICOM [29,31] and JPEG formats and are accompanied by metadata (Fig. 1). This metadata includes patient ID, gender, age, and the general anatomical location, along with the target value. The training dataset consists of two categories of images: benign and malignant, each labelled with its ground truth.

For the melanoma classification study conducted using the ISIC datasets, the metadata and image distribution across the training, test, and validation sets are outlined in a Tab. 1. The dataset maintains an approximate 70%-30% split between benign and malignant cases. Of the 17,114 images, 14,204 were allocated to training, while 1,455 images were allocated for testing and an additional 1,455 for validation.

Data preprocessing

To ensure the effectiveness of the melanoma detection model, a comprehensive data pre-processing pipeline was implemented. The primary goal of the preprocessing was to standardize and enhance the input images to improve the model's ability to generalize to unseen data while mitigating overfitting. The preprocessing was tailored to each model paying particular attention to the size of the input image sizes.

During the training phase, a sequence of dynamic, on-the-fly transformations was employed to virtually increase data diversity without altering the static size of the dataset or the class distribution. This approach ensures that the model encounters slightly different versions of the training images in each epoch [32].

The input images were first resized to fit the requirements of each architecture. To simulate real-world clinical conditions, such as varying handheld dermatoscope orientations, random horizontal and vertical flips were applied, followed by random rotations of up to 30 degrees. This specific rotation range was chosen to introduce sufficient variability while preventing excessive information loss or artifacts caused by padding and cropping, which could otherwise obscure critical diagnostic features like irregular lesion borders. Additionally, color jittering was applied to adjust brightness and contrast, simulating lighting variations during image acquisition. Finally, images were normalized using ImageNet's mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225] to facilitate stable and efficient training.

These preprocessing steps not only standardized the input data, but also introduced controlled variability into the training data set. This ultimately improved the model’s robustness and performance of the model in melanoma detection tasks.

Deep learning architectures

The fundamental reason for selecting these three specific models was their complementarity. Each architecture interprets skin lesions differently based on its unique design. By combining them into an ensemble using the soft voting method, we successfully mitigated individual model weaknesses and leveraged their collective strengths.

Ensemble Learning

Ensemble learning is a powerful machine learning paradigm that combines the predictions of multiple models to improve overall performance and robustness. Ensembles, especially when individual models have complementary strengths, often achieve higher accuracy than individual models. They are also more robust to data outliers and noise, as errors in individual models are often corrected by others. There are several types of ensemble techniques, including voting, stacking, bagging and boosting [38,39]. Weighted majority voting (also known as soft voting) and majority (hard) voting methods to aggregate predictions from the ensemble were applied in this study (Fig. 5).

Fig. 5. Example of an ensemble learning with hard and soft voting

In hard voting, the final prediction is determined by a majority vote, with each model contributing equally to the decision. In contrast, soft voting averages the predicted probabilities from all models, using confidence scores to produce a more nuanced and potentially more accurate result, especially for unbalanced datasets [39]. These techniques capitalize on the complementary strengths of individual models, ensuring a well-rounded and robust performance in dermoscopic image classification.

The mathematical formula for voting [40]:

where: K stands for the number of classes, \omega_i represents the weight assigned to the i-th model m_i, m_i^j(x) is the output of the i-th model for the j-th category label. Hard voting can be readily modeled by assigning the same positive value to each individual weight \omega_i such as setting \omega_i=1 for all models.

Two main types of voting approaches exist: hard voting and soft voting. In hard voting, each model’s prediction counts as a single vote, and the final output is determined by the majority or plurality of votes. While straightforward, hard voting can disregard useful information about classifier confidence. Soft voting, on the other hand, integrates class probabilities or confidence scores, weighting each classifier’s vote according to how certain it is about its prediction. This approach can lead to more nuanced ensemble decisions, particularly for datasets with overlaps or class imbalances.

In addition to evaluating the performance of individual models, ensemble methods such as soft voting, hard voting, and stacking were applied to further enhance classification performance. The ensemble was composed of diverse deep learning architectures trained independently on the same dataset. In case of soft voting, the predicted class probabilities from each base model were averaged to produce the final prediction, allowing the ensemble to reflect the relative confidence of each model. For hard voting, the final class was selected based on the majority class label predicted by each model.

Stacking was also explored as a higher-level ensemble strategy. In this approach, the outputs of multiple base learners are combined using a meta-learner trained to optimize the final prediction. The XGBoost algorithm, an efficient and scalable implementation of gradient boosting, was selected as the meta-classifier due to its high predictive performance and ability to model complex feature interactions. XGBoost builds an ensemble of decision trees in a sequential manner, where each new tree corrects the errors made by the previous ones, and is particularly effective in handling structured data and reducing overfitting through regularization [41].

In this study, XGBoost was employed as a meta-classifier in a stacking ensemble framework. Specifically, the output probabilities obtained by trained ResNet152, DenseNet201, and EfficientNet-B4 models were used as input features for the XGBoost model, which learned to optimize the final prediction by leveraging the individual strengths and compensating for the weaknesses of each base model. This hierarchical ensemble strategy aimed to improve generalization and overall classification accuracy by combining diverse model architectures. The predictive performance of the stacked model was compared against that of the individual CNNs to evaluate the added value of this ensemble learning approach.

Evaluation metrics

The trained model’s performance was evaluated using balanced accuracy (2), precision (3), recall (4), and F1 score (5) on the test dataset. Due to the imbalanced nature of the dataset, accuracy was used. The other metrics were calculated using the parameter average=weighted to account for class imbalance. True Positives (TP) and True Negatives (TN) denote the number of correctly predicted subjects, while False Positives (FP) and False Negatives (FN) correspond to the incorrectly predicted subjects.

Experimental Setup

All experiments were conducted on the same hardware setup to ensure consistency and reproducibility. Model training was carried out entirely on local equipment, allowing for greater control over the process.

The hardware specifications used for training are as follows:

This setup provided sufficient computational power for training moderately sized deep learning models while addressing the challenges posed by memory limitations.

The preparation of the study environment involved installing Python version 3.11, along with essential libraries such as PyTorch (version 2.4.1), NumPy (version 1.23.5), and Pandas (version 2.2.2). Additionally, libraries for image processing and result visualization, such as Matplotlib (version 3.9.2), were imported. The environment configuration also included setting the GPU as the primary device for model training, enabling faster computation and parallel processing to speed up the learning process.

Training models

The model training process began with data pre-processing, which included image resizing and normalization. The choice of image size was determined by the selected architecture, 224x224 for ResNet152 and DenseNet201, and 380x380 for EfficientNet-B4. The pre-trained architectures, adapted for the skin image classification task by tuning the final network layers, were trained on 14,204 images from the ISIC database. To address the dataset imbalance, a cost-sensitive learning approach was adopted by applying class weights to the loss function. The weights were calculated based on the inverse frequency of each class (1.44 for benign and 3.29 for malignant), ensuring that the minority class significantly influenced the model’s optimization process. The training was conducted using the AdamW optimizer with weight decay to further enhance regularization and prevent overfitting. The training process was monitored using callbacks such as EarlyStopping, which halts training when the model's performance stops improving, and weight decay regularization, which helps to prevent overfitting by penalizing large weights during training. These parameters, summarized in Table 2, were selected based on preliminary tuning to maximize the accuracy and stability of each model. Additionally, the technique of freezing and unfreezing layers in pre-trained models was applied; however, it led to performance improvement only in the case of EfficientNet-B4.

Following the training process, the models were tested on a separate test set of 1,455 images. Testing the model involves assessing its ability to correctly diagnose melanoma based on previously unknown data. Each image was classified as ‘malignant’ or ‘benign’ and the results were saved for later analysis.

The final step of the study involved comparing the performance of the models created using ensemble learning techniques with other popular CNN architectures, such as ResNet152, DenseNet201, and EfficientNet-B4. This analysis aimed to determine which architecture performs best in the automatic diagnosis of cutaneous melanoma. Additionally, the performance of the ensemble learning model was compared to that of the individual models to assess the potential advantages of combining multiple architectures.

Visualization technique: Grad-CAM

 To enhance the transparency and trustworthiness of the proposed diagnostic system, Gradient-weighted Class Activation Mapping (Grad-CAM) was employed. This technique produces a localized heatmap by utilizing the gradients of a specific target class (e.g., 'Malignant') flowing into the final convolutional layer of the network [42]. By calculating the importance of each neuron through global average pooling of the gradients, Grad-CAM highlights the specific pixels and regions that most significantly influenced the model's prediction.

In this study, Grad-CAM was applied to each individual base model (ResNet152, DenseNet201, and EfficientNet-B4) rather than the ensemble as a whole. This approach allows for a granular analysis of how different architectures perceive dermatological features. By verifying that each network in the ensemble focuses on clinically relevant areas, such as irregular borders or pigment distribution, we ensure that the final combined decision (via soft voting) is based on meaningful medical patterns rather than image artifacts.

Single deep learning architectures studies

Plots of loss and accuracy curves for each model provide visual insight into the training process, illustrating stability and rate of convergence (Fig. 6-8). The graphs provide a better understanding of the performance of each architecture in the melanoma identification task. In addition, these plots allow one to see the differences that occur during training due to the use of the layer freezing technique of the pre-trained model.

Analysis of the plots indicates that all models learn efficiently, and that the differences between loss and accuracy on the training and validation sets are within acceptable limits. In the early epochs, we observe a significant reduction in loss for both the training and validation sets, indicating that all models are learning effectively. As training progresses, the differences between the loss for the validation and training sets become small, indicating good generalization of the model. The small differences in accuracy between the test set and the training set indicate an appropriate balance between learning and generalization. In case of the EfficientNet-B4 model, the loss and accuracy curves have the most dynamic start due to the layer freezing technique and its unfreezing after the 10th epoch of training. However, in the further stages of training, the differences decrease and the accuracy on the test set reaches a high level. The higher test accuracy compared to training in some epochs may be indicative of the effectiveness of the regularization used in this particular model of the model.

After the learning process, each model was evaluated to assess its performance and reliability. Model performance was determined by making predictions on test data and comparing the results to actual labels (Tab. 3). To ensure the robustness and consistency of the evaluation, each model was tested across 10 independent runs, with performance metrics averaged to account for variability and improve result reliability. The comprehensive evaluation performed on the test dataset yielded insightful results, with the performance metrics for each model summarized in Tab. 3. In addition, Fig. 9, 10, and 11 present the corresponding confusion matrix, providing a detailed breakdown of each model’s classification performance. Tables 4, 5, and 6 further present the inference results for the trained models, with a breakdown for each class.

This study identified EfficientNet-B4 as the optimal pre-trained model for melanoma classification, providing the highest accuracy and stability on the test dataset. These results highlight the potential of deep learning models to improve diagnostic accuracy and assist dermatologists in detecting malignant melanoma lesions.

The analysis performed on the test set (Tab. 4-6) provided detailed results, comparing the performance metrics of different models to assess their effectiveness in melanoma classification. EfficientNet-B4 outperformed all other models in key metrics, achieving an F1-score of 90.6%, indicating an excellent balance between precision (90.8%) and sensitivity (90.4%). This confirms its effectiveness in classifying melanoma as malignant or benign, as it identifies positive cases exceptionally well as true positives, effectively detecting malignant melanoma lesions. ResNet152 performed similarly but slightly lower, while DenseNet201 had the weakest performance in each category, although it also achieved high scores. The small differences in results between the models highlight the superiority of EfficientNet-B4 in analyzing complex dermoscopic patterns.

Analysis of confusion matrix (Fig. 9-11) reveals that all models have a high number of correct predictions for the 'Benign' class. The main difference between the models is the number of misclassified malignant lesions (false negatives). EfficientNet-B4 recorded 51 false negatives (FN), indicating a need for further optimization, particularly for melanoma diagnosis. Based on the results for each class (Tab. 4-6), the average precision for the 'Benign' class was 95 ± 1%, demonstrating all models' ability to accurately diagnose benign lesions. Conversely, for the 'Malignant' class, the precision was lower, 78 ± 4%, highlighting the challenge of diagnosing malignant lesions, which often have subtle and variable characteristics. The F1-score for the 'Malignant' class averaged 83 ± 2%, indicating a good balance between precision and sensitivity, although there is still potential for improvement.

The results, gathered in Tab. 4-6, suggest that the EfficientNet-B4 architecture, with frozen layers during training, is the most suitable for skin cancer identification, combining high precision with exceptional stability. This model is particularly notable for its ability to balance precision and recall for malignant cases (the minority class), making it the most reliable model for this task.

Ensemble learning studies

This study focuses on evaluating how various ensemble learning techniques influence on enhancing the performance of melanoma skin cancer detection. The soft voting, hard voting and XGBoost methods were compared. The use of a soft-voting approach, which combines the probabilities predicted by different models, produced better results than individual models (Tab. 7). This approach allows the strengths of each model to be exploited.

The peak accuracy of 91.3% was shared by two ensemble variants: the combination of all three models and the pairing of ResNet152 with EfficientNet-B4. Notably, the latter combination proved to be slightly more robust, yielding a lower standard deviation of ±0.06 compared to ±0.08 for the three-model ensemble, suggesting that a two-model synergy is sufficient to achieve optimal results in this task. The combination of ResNet152 and EfficientNet-B4 also achieved an accuracy of 91.3%, suggesting that these models classify exceptionally well together. The ensemble of DenseNet201 and EfficientNet-B4 achieved an accuracy of 91.0%, which is slightly lower, but still indicates a very good generalization. On the other hand, the combination of ResNet152 and DenseNet201 resulted in an accuracy of 89.5%, which is the lowest of the ensembles tested. However, it still shows a slight improvement over the individual models, by 0.3% and 1.2% for ResNet152 and DenseNet121, respectively. All variants of the ensembles outperformed the accuracy of the individual models, highlighting the advantages of ensemble learning in the classification of dermoscopic images.

In contrast to the results obtained with soft voting, the performance of the ensemble models with hard voting and the XGBoost stacking ensemble is noticeably lower across all metrics (Tab. 8-9). Only combination of all deep learning models, or DenseNet201 together with EfficientNet-B4 with hard voting provided improvements according to the single architecture performance. In the case of XGBoost this technique is stated to slightly improve skin cancer detection according to DenseNet201.

These results clearly demonstrate the superiority of soft voting for skin cancer classification tasks. The improved performance of soft voting is likely due to its ability to leverage the probabilistic outputs of individual models. This allows for a more nuanced and effective combination of predictions [43]. In contrast, both hard voting and the XGBoost stacking ensemble, which rely on discrete or weighted decisions, appear to be less effective at capturing the complementary strengths of individual models, resulting in lower accuracies and balanced accuracies. This further highlights the importance of using soft voting in ensemble learning to achieve optimal results in dermoscopic image classification.

Explainable of deep learning models

Machine learning models used for melanoma detection often leverage visualization techniques like Grad-CAM. These techniques help understand which areas of an image influence the model's decisions.

The ResNet152 model primarily focuses on the central part of the skin lesion, particularly areas with the highest color intensity (Fig. 12). It emphasizes the dark center while largely ignoring the surrounding skin. This suggests that the model effectively isolates key diagnostic regions, making it particularly useful for detecting lesions with well-defined borders and intense pigmentation.

The DenseNet201 model distributes its attention more evenly across the lesion, a pattern commonly observed in benign lesions with regular shapes and uniform color (Fig. 13). Its broader focus suggests an analysis of the lesion’s overall context, including surrounding tissue. The absence of sharply highlighted areas indicates lesion consistency, making this model suitable for identifying lesions with unclear boundaries that require a more comprehensive assessment.

The EfficientNet-B4 model concentrates on the lesion’s irregular edges and darker regions, features characteristic of melanoma, which often presents with asymmetry, irregular borders, and variable pigmentation (Fig. 14). This model exhibits a more complex pattern of interest, analyzing multiple features simultaneously. Such an approach is valuable for detecting intricate and irregular lesions that require multifaceted evaluation.

Each of these models effectively identifies key diagnostic regions in skin lesions, though their interpretations vary based on architectural differences. Grad-CAM visualizations not only provide insights into model performance but also enhance interpretability.

Training stability and generalization

A critical challenge in deep learning for medical imaging is maintaining an optimal balance between learning complex patterns and ensuring generalization. In this study, the risk of overfitting was mitigated through a combination of weight decay regularization and EarlyStopping callbacks, which prevented the networks from over-optimizing on the training data. Furthermore, the use of dynamic data augmentation simulated real-world variability, ensuring that the models remained robust to noise and minor image artifacts. As shown in the training curves (Fig. 6-8), the convergence of training and validation metrics indicates that the models successfully avoided significant overfitting. Conversely, the risk of underfitting was addressed by employing powerful pre-trained architectures and fine-tuning their deep layers to capture the subtle, irregular features characteristic of malignant melanoma. The ensemble approach further stabilized these results, as the soft voting mechanism helped compensate for individual architectural biases.

Comparison with the state-of-the-art

The state-of-the-art studies (Tab. 10) provide an in-depth insight into skin cancer identification utilizing various types of CNN and deep learning architectures. The studies performed using the ISIC datasets were only considered. The majority of papers involve the single models. The pre-trained architectures assure accuracy between 71.19% for VGG-19 and 90.40% for the EfficientNet-B4. Among the common deep learning architectures, ResNet-based models are the most applied for melanoma detection. However, various CNN-based solutions provide quite good performance that vary between 71% and 97.41% accuracy, depending on their configurations and hyper parameters. The minority of studies concern ensemble learning for melanoma identification. Three types of ensemble learning methods, hard voting, soft voting, and stacking, were analyzed. The highest performance, 93% accuracy, was obtained for the stacking model. It should be emphasized that ensemble learning is a promising direction of studies in enhancing skin cancer identification. That is why the studies presented in this paper are of great values.

This study highlights the great potential of advanced deep learning models in classifying skin lesions for melanoma diagnosis, improving accuracy of detection. The results show that deep learning models such as ResNet152, DenseNet201 and EfficientNet-B4 achieve high accuracy in melanoma identification. In particular, the application of an ensemble learning technique, using soft-voting combining all three architectures, improved the precision rates and F1-score up to 91.3%.

In addition, soft-voting proved to be much more effective on an unbalanced dataset than the hard-voting technique. In contrast, the stacking method using XGBoost did not produce the expected improvement in performance, suggesting that this technique may be less effective for dermoscopic images.

The results align with existing literature (Tab. 10), highlighting the high performance of EfficientNet models in melanoma diagnosis, with precision and F1-scores exceeding 85%. Similar to this study, other research has demonstrated improved classification rates using ensemble learning techniques, particularly voting, which confirms the accuracy of this approach. Conversely, the results for stacking suggest that this method may not be optimal for classifying highly complex images. This study confirms the effectiveness of deep learning models and ensemble learning techniques in skin melanoma classification. In particular, EfficientNet-B4 models and the use of an ensemble model with soft-voting proved to be particularly effective, achieving higher precision and F1-scores than individual models.

The results of this study successfully address the formulated research questions. While all models showed high competence, EfficientNet-B4 emerged as the most reliable single architecture, achieving 90.40% accuracy. Its success is attributed to its compound scaling and SE-blocks, which effectively capture asymmetric borders. For research question 2 and 3, our findings demonstrate that ensemble learning, specifically soft voting, significantly enhances performance, reaching an absolute accuracy of 91.30%. This confirms that leveraging probabilistic confidence scores is more effective than the discrete logic of hard voting or the meta-classification approach of XGBoost stacking in this domain.