MALWARE DETECTION SYSTEM BASED ON DEEP LEARNING TECHNIQUE

Abstract

In this paper, we propose a malware classification and detection framework using transfer learning based on existing Deep Learning models that have been pre-trained on massive image datasets, we applied a deep Convolutional Neural Network (CNN) based on Xception model to perform malware image classification. The Xception model is a recently developed special CNN architecture that is more powerful with less overfitting problems than the current popular CNN models such as VGG16, The experimental results on a Malimg Dataset which is comprising 9,821 samples from 26 different families ,Malware samples are represented as byteplot grayscale images and a deep neural network is trained freezing the convolutional layers of Xception model adapting the last layer to malware family classification , The performance of our approach was compared with other methods including KNN, SVM, VGG16 etc. , the Xception model can effectively be used to classify and detect  malware families and  achieve the highest validation accuracy  than all other approaches including VGG16 model which are using image-based malware, our approach does not require any features engineering, making it more effective to adapt to any future evolution in malware, and very much less time consuming than the champion’s solution.