START macOS Internals for Detection Eng

Exploring macOS system APIs for detection engineering

Once you’re comfortable with Objective-C syntax and basics, you can start exploring the macOS-specific APIs that are relevant to monitoring and security. Apple provides frameworks (some Objective-C, some C) for interacting with system internals.

Filesystem monitoring

Historically, macOS has provided the FSEvents API (in <Foundation/NSFileManager> or via the <fcntl.h> for kqueue) to watch file system changes, and the OpenBSM audit system for detailed event logs. These older methods are C APIs, but you can invoke them from Objective-C.

However, Apple’s current approach is the Endpoint Security Framework (ESF), introduced in macOS 10.15 (Catalina). ESF allows user-space applications to subscribe to a wide range of system events (e.g., process execs, file writes, network, etc.) in real-time. It’s a C API, but you typically write an Objective-C or Swift program to use it. Using ESF does require a special entitlement from Apple (and a signed app), which can be a hurdle.

If you’re just experimenting, you can run ESF clients on a Mac with System Integrity Protection (SIP) disabled, or target only your own machine for now (demo code idea). ESF is powerful (also noisy, haha): it gives you structured events and can even block actions (like an EDR product would do).

Apple has sample code and WWDC videos on ESF (e.g., “Monitoring System Events with Endpoint Security”_ and _“Build an Endpoint Security App”). Objective-See’s blog has a great walkthrough of building a file monitor with ESF.

Process monitoring

If you want to detect process executions or inspect running processes, you have a few options. The ESF (mentioned above) provides process exec events (ES_EVENT_TYPE_NOTIFY_EXEC etc.), which is ideal.

Without ESF, you can use NSWorkspaceDidLaunchApplicationNotification (part of AppKit) to be notified when a new app is launched, but that only catches applications (not all processes) and is more high-level. Another way is using the POSIX APIs: e.g., periodically calling sysctl or procinfo() to get process lists (tools like OSQuery use such methods internally).

If you are comfortable with C, you could also use fork/exec hooking in a dylib or writing a LaunchDaemon that uses wait4() to watch for child processes, but those are less straightforward.

For reverse engineering or analysis, know that macOS processes can be examined via the ps and launchctl commands (for listing running jobs and launch items), and programmatically via the ProcessManager (old Carbon) or NSRunningApplication class. Try using [NSRunningApplication currentApplication] and related methods to enumerate running apps, or use NSTask to spawn system commands like /bin/ps and parse output, as a simple route.

Interacting with hardware (mic, camera, etc.)

macOS provides frameworks like AVFoundation for camera/mic, and IOKit for lower-level hardware events. For example, detecting microphone usage might involve hooking into the audio input APIs or using IOKit to receive hardware change notifications. An example of a security tool in this space is Objective-See’s OverSight, which monitors mic and webcam usage (likely by listening for specific system notifications or API calls). If you’re curious, you can read its source code to see how it uses Objective-C to subscribe to those events.

User activity & privacy events

Some detections involve catching things like keystroke events (keylogger detection) or screenshot attempts. These often require accessibility permissions and use APIs like Quartz Event Taps (C API) or NSAccessibility (Objective-C). For instance, a simple keylogger can be made with Quartz event taps in C, but you could wrap that in an Objective-C app. Understanding how legitimate apps use these APIs lets detect misuse. Apple’s Security & Privacy guide and documentation on Accessibility APIs can be resources if you go down that route.

As you explore these, Apple’s Developer Documentation is your friend. In Xcode’s documentation viewer, search for classes like EndpointSecurity, NSWorkspace, FSEventStream, NSNotificationCenter, IOKit etc., to read the official guides and reference. Many APIs will have sample code in the docs or on the Apple Developer site. Additionally, WWDC session videos often give a concise overview of how to use these frameworks in code (for ESF, check WWDC 2019/2020 sessions; if you can’t find documentation from Apple, search The Eclectic Light company blog).