Abstract: Direct kernel object manipulation(DKOM) attacks hide the kernel objects through direct access and modification to the kernel objects.Such attacks are a long-term critical security issue in mainstream operating systems.The behavior-based online scanning can efficiently detect limited types of DKOM attacks,and the detection procedure can be easily affected by the attacks.In recent years,memory-forensics-based static analysis has become an effective and secure detection approach in the systems potentially attacked by DKOM.The state-of-the-art approach can identify the Windows system kernel objects using a graph neural network model.However,this approach cannot be adapted to Linux kernel objects and has limitations in identifying small kernel objects with few pointer fields.This paper designs and implements a deep-learning-based DKOM attack detection approach for Linux systems to address these issues.An extended memory graph structure is proposed to depict the points-to relation and the constant fields' characteristics of the kernel objects.This paper uses relational graph convolutional networks to learn the topology of the extended memory graph to classify the graph nodes.A voting-based object inference algorithm is proposed to identify the kernel objects' addresses.The DKOM attack is detected by comparing our kernel object identification results and the results of the memory forensics framework Volatility.The contributions of this paper are as follows.1) An extended memory graph structure that improves the detection effectiveness of the existing memory graph on capturing the features of small kernel data structures with few pointers but with evident constant fields.2) On the DKOM attacks raised by five real-world Rootkits,our approach achieves 20.1% higher precision and 32.4% higher recall than the existing behavior-based online scanning tool chkrootkit.

Key words: Memory forensics, Malware detection, Operating system security, Graph neural network, Binary analysis