Enhancing Malware Detection by Integrating Machine Learning with Cuckoo Sandbox
Article content
Abstract
In this work, two categories of deep learning and conventional machine learning were used to classify malware using a dataset of all possible API call sequences. Specifically, the objective was to determine the best strategy to tackle the ever-rising menace as malware becomes more complex. A new dataset was created employing Cuckoo Sandbox, where API call sequences originating from both benign and malware samples were recorded. The performance of these algorithms was benchmarked and tested using this dataset, which includes SVM, RF, KNN, XGB, GBC, CNN, and RNN. The study established that both deep learning and conventional machine learning algorithms provided high accuracy above 90%. Specifically, the recurrent neural networks (RNNs) demonstrated high accuracy rates ranging from 95% to 99%. These results are highly indicative of deep learning, especially RNN, as a promising approach to improving the effectiveness of malware detection. The data obtained from dynamic analysis, when integrated into a database, serves as a more reliable source for training and testing of such models, and can improve the model’s ability to identify new threats posed by malware. Thus, this work is salient in enhancing the development of new approaches to fight malware that constantly evolve in the modern world.
Keywords
Malware Analysis, Machine Learning, Deep Learning, Malware Dataset, Cuckoo Sandbox, API call sequencing
Downloads
References
M. Ali, S. Shiaeles, N. Clarke, and D. Kontogeorgis, "A proactive malicious software identification approach for digital forensic examiners," Journal of Information Security and Applications, vol. 47, pp. 139–155, Aug. 2019, doi: https://doi.org/10.1016/j.jisa.2019.04.013.
C. Li, Q. Lv, N. Li, Y. Wang, D. Sun, and Y. Qiao, "A novel deep framework for dynamic malware detection based on API sequence intrinsic features," Computers & Security, vol. 116, p. 102686, May 2022, doi: https://doi.org/10.1016/j.cose.2022.102686.
AV-TEST, "Malware Statistics & Trends Report," Av-test.org, Apr. 25, 2019. [Online]. Available: https://www.av-test.org/en/statistics/malware/. [Accessed: 20-May-2022].
M. Sahin and S. Bahtiyar, "A Survey on Malware Detection with Deep Learning," in Proc. 13th Int. Conf. Security of Information and Networks, Nov. 2020, doi: https://doi.org/10.1145/3433174.3433609.
A. Amira, A. Derhab, E. B. Karbab, and O. Nouali, "A survey of malware analysis using community detection algorithms," ACM Computing Surveys, vol. 56, no. 2, pp. 1–29, Sep. 2023, doi: https://doi.org/10.1145/3610223.
S. Talukder and Z. Talukder, "A Survey on Malware Detection and Analysis Tools," Int. J. Network Security & Its Applications, vol. 12, no. 2, pp. 37–57, Mar. 2020, doi: https://doi.org/10.5121/ijnsa.2020.12203.
M. Y. Wong, M. Landen, M. Antonakakis, D. M. Blough, E. M. Redmiles, and M. Ahamad, "An Inside Look into the Practice of Malware Analysis," in Proc. 2021 ACM SIGSAC Conf. Computer and Communications Security, Nov. 2021, doi: https://doi.org/10.1145/3460120.3484759.
B. Yadav and S. Tokekar, "Deep Learning in Malware Identification and Classification," in Malware Analysis Using Artificial Intelligence and Deep Learning, M. Stamp, M. Alazab, and A. Shalaginov, Eds. Cham: Springer, 2021, doi: https://doi.org/10.1007/978-3-030-62582-5_6.
R. Patil and W. Deng, "Malware Analysis using Machine Learning and Deep Learning techniques," in Proc. 2020 SoutheastCon, Mar. 2020, doi: https://doi.org/10.1109/southeastcon44009.2020.9368268.
J. Singh and J. Singh, "Detection of malicious software by analyzing the behavioral artifacts using machine learning algorithms," Information and Software Technology, vol. 121, p. 106273, May 2020, doi: https://doi.org/10.1016/j.infsof.2020.106273.
A. M. Lajevardi, S. Parsa, and M. J. Amiri, "Markhor: malware detection using fuzzy similarity of system call dependency sequences," J. Computer Virology and Hacking Techniques, Apr. 2021, doi: https://doi.org/10.1007/s11416-021-00383-1.
F. O. Catak, A. F. Yazı, O. Elezaj, and J. Ahmed, "Deep learning based Sequential model for malware analysis using Windows exe API Calls," PeerJ Computer Science, vol. 6, p. e285, Jul. 2020, doi: https://doi.org/10.7717/peerj-cs.285.
S. Li, Q. Zhou, R. Zhou, and Q. Lv, "Intelligent malware detection based on graph convolutional network," The Journal of Supercomputing, Aug. 2021, doi: https://doi.org/10.1007/s11227-021-04020-y.
"MalwareBazaar | Malware sample exchange," bazaar.abuse.ch. [Online]. Available: https://bazaar.abuse.ch/. [Accessed: 10-Feb-2023].
T. Copyright, "Latest entries - The Portable Freeware Collection," www.portablefreeware.com. [Online]. Available: https://www.portablefreeware.com/. [Accessed: 10-Feb-2023].
J. Palša et al., "MLMD—A Malware-Detecting Antivirus Tool Based on the XGBoost Machine Learning Algorithm," Applied Sciences, vol. 12, no. 13, p. 6672, Jul. 2022, doi: https://doi.org/10.3390/app12136672.
H. S. Gill, O. I. Khalaf, Y. Alotaibi, S. Alghamdi, and F. Alassery, "Multi-Model CNN-RNN-LSTM Based Fruit Recognition and Classification," Intelligent Automation & Soft Computing, vol. 33, no. 1, pp. 637–650, 2022, doi: https://doi.org/10.32604/iasc.2022.022589.
A. Oliveira, "Malware Analysis Datasets: API Call Sequences," www.kaggle.com. [Online]. Available: https://www.kaggle.com/datasets/ang3loliveira/malware-analysis-datasets-api-call-sequences. [Accessed: 20-Jan-2023].
Article details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
You must read and accept the copyright terms and conditions(click here for copyright terms).