Introduction to Windows System Calls: Exploring OS Interaction in Brief

To keep the previous one concise, this article will delve deeper into the realm of syscalls. For a foundational understanding of Windows system calls, refer to this article as a primer. :)

System calls are fundamental mechanisms that allow programs to interact with the operating system kernel, requesting essential services such as file I/O, memory allocation, and hardware communication. NTDLL.DLL, in turn, plays a crucial role in the Windows environment, serving as a dynamic-link library (DLL) that provides access to Windows NT system calls. It acts as an intermediary layer between user-mode applications and the operating system kernel, offering a variety of functions that programs can invoke to perform specific system operations. EDRs hook into system calls in NTDLL.DLL because it serves as the primary interface between user space and the kernel, enabling the detection and real-time response to suspicious activities. The figure below illustrates how NTDLL.DLL acts as an interface between application requests from user services and the system kernel.

In Windows, each syscall is identified by a unique number, known as a syscall number. Within the NTDLL.DLL library, these syscall numbers are sequentially generated, providing a numerical order for the various system functionalities. The System Service Descriptor Table (SSDT) is a kernel data structure that maps syscall numbers to their corresponding functions within the kernel. It serves as a translation table, enabling the operating system to efficiently direct syscall invocations to the appropriate kernel routines. In 64-bit systems, the system call number is passed via the RAX register, while parameters follow the Microsoft Calling Convention. Examining NTDLL.DLL in IDA Pro, we find NtProtectVirtualMemory’s system call number is 0x50, which may vary across Windows versions.

The syscall instruction is a low-level instruction used in x86 and x86–64 architectures (Intel and AMD processors) to initiate a system call within the operating system. It serves as the primary mechanism through which user-level applications can request services or functionality provided by the operating system kernel.