A novel technique for ransomware detection using image based dynamic features and transfer learning to address dataset limitations.

The increasing frequency of ransomware attacks necessitates the development of more effective detection methods. Existing image-based ransomware detection approaches have largely focused on static analysis, overlooking specialized ransomware behaviors such as encryption, privilege escalation, and system recovery disruption. Although dynamic and memory forensics-based visualization methods exist in the broader malware domain, they primarily target generic malware families and often rely on memory dumps or system snapshots without transforming behavioral features into spatially meaningful representations. Moreover, traditional machine learning methods such as Random Forest (RF), Support Vector Machine (SVM), and K-Nearest Neighbors (KNN) typically depend on manual feature engineering and large labelled datasets, limiting scalability and adaptability. To address these limitations, we propose a novel behavior-to-image ransomware detection framework that transforms dynamic behavioral features extracted from sandbox-generated JSON reports into two-dimensional (2D) grayscale and color image representations, optimized for transfer learning (TL), enabling effective classification under small-data conditions. Our approach integrates domain-specific feature filtering and impact analysis to ensure the selection of the most ransomware-relevant attributes. TL subsequently automates feature extraction and classification, eliminating the need for separate feature selection procedures and overcoming the time-consuming process of manual feature engineering. Furthermore, by leveraging prior knowledge from large-scale image datasets, TL significantly mitigates the need for extensive labelled data while maintaining high detection accuracy and strong generalization. Experimental results demonstrate that fine-tuned pretrained models, notably ResNet50, achieve up to 99.96% accuracy with a minimal loss factor of 0.0026, even with a small dataset of 500 ransomware and 500 benign samples. We further validated the model's interpretability through t-SNE visualizations and saliency maps, confirming its ability to focus on class-discriminative behavioral patterns. The low misclassification rate, along with the transparency of the model, highlights its potential for practical deployment in ransomware detection systems.