Abstract
The widespread expansion of Internet of Things devices has ushered in an era of unprecedented connectivity. However, it has simultaneously exposed these resource-constrained systems to novel and advanced cyber threats. Among the most impressive and complex attacks are those leveraging in-memory shellcode runners (malware), which perform malicious payloads directly in memory, circumventing conventional disk-based detection security mechanisms. This paper presents a comprehensive framework, both academic and technical, for detecting in-memory shellcode runners, particularly tailored to the unique characteristics of these networks. We analyze and review the limitations of existing security parameters in this area, highlight the different challenges posed by those constraints, and propose a multi-layered approach that combines entropy-based anomaly scoring, lightweight behavioral monitoring, and novel Graph Neural Network methods for System Call Semantic Graph Analysis. Our proposal focuses on runtime analysis of process memory, system call patterns (e.g., Syscall ID, Process ID, Hooking, Win32 application programming interface), and network behavior to identify the subtle indicators of compromise that portray in-memory attacks, even in the absence of conventional file-system artifacts. Through meticulous empirical evaluation against simulated and real-world Internet of Things attacks (red team engagements, penetration testing), we demonstrate the efficiency and a few challenges of our approach, providing a crucial step towards enhancing the security posture of these critical environments.