ROBUST ANDROID MALWARE DETECTION AGAINST OBFUSCATION AND ADVERSARIAL ATTACKS USING RGB MARKOV IMAGES AND DEEP ENSEMBLE LEARNING

Abstract

Android malware detection faces persistent challenges as attackers increasingly employ obfuscation and adversarial manipulation to evade conventional static and signature-based defenses. Existing detectors often fail to generalize when code structures are altered or encrypted, which results in poor resilience against real-world evasion. This thesis introduces a deep ensemble framework that transforms Android APK components into RGB Markov images, capturing both structural and statistical byte-level patterns. The ensemble integrates EfficientNet-B0, ConvNeXt-Small, and Swin-Base architectures, combining their predictions through majority voting to ensure stable and reliable detection. A balanced dataset, KindiDroid, comprising 95,400 images, including unobfuscated and 13 Obfuscapk-based variants, was constructed for evaluation. The framework achieved an F1-score of 99.13% and an AUC of 99.86% on clean data, maintaining over 96% accuracy across diverse obfuscation techniques. After adversarial training, robustness surpassed 97% under FGSM attacks on both clean and obfuscated inputs, demonstrating strong generalization against black-box and white-box threats.