Objective-See

Monitoring Process Creation via the Kernel (Part III)

12/13/2015

The previous two blog posts discussed why BlockBlock required processes creation notifications, and showed several ways to achieve this via a kernel extension. Today, let's conclude this blog mini-series by describing one way to get this 'process creation information' from the kernel to a user-mode application.

The starting point for this blog is in the kernel, where the kext has just received a process creation notification, either via a MAC policy or a KAuth listener. This information must now be delivered to the user-mode component of BlockBlock so that when a persistent file I/O event occurs, the event may be correlated with the responsible process.

Kernel-Mode -> User-Mode Communication Options
There are several ways to communicate between kernel-mode and user-mode. In "Boundary Crossings" Apple mentions methods such as Mach messages, RPC-Mig, BSD syscalls, and memory mapping. A stackoverflow posting describes others, such as IOUserClient, sockets, and the Kernel Control API (ctl_enqueuedata). Which one to choose!?

Discussing the pros and cons of all these various communication mechanisms is beyond the scope of this blog post. However, while some seem overly complex (RPC-Mig), others are KPIs or (now) not recommended (e.g. sysctl/ioctls). Finally, others are simply not applicable, such as IOUserClient (since BlockBlock's kext does not make use of IOKit).

In the end I decided to go with a system socket. This method proved simple, and IMHO, a reliable way to deliver data from a kext to user-mode. In short, via the kern_event API, the kext could broadcast events (e.g. process creations) that could be recv'd via the socket in user-mode. Of course due to Apple's limited (and incorrect!) documentation, this took a decent amount of effort and spelunking around to get it working properly. However, all's well that ends well - so no hard feelings ;)

Broadcasting Data via the Kernel Events API
In Apple's "Network Kernel Extensions Programming Guide," there's a section that describes how to "receive broadcast [data] from the kext." Specifically it states that one can post a notification (with data) to a user-mode system socket, via the "kev_message_post [function] with a kev_msg structure." Ok, so let's google "kev_message_post" to find out more about this function.

Two hits: the aforementioned NKE programming guide, and a posting to Apple's mailing list in 2006 which states:

"However it [Apple's documentation] doesn't provide any examples or documentation for using said API, and a Google search for "kev_message_post" returns exactly one result: the same document on Apple's web site."

No info on this function? Moreover, no function in the kernel's disassembly matches kev_message_post:

Hrmmm.....

Well, after more googling, it turns out the function is in reality named kev_msg_post, not kev_message_post. Jeez, thanks Apple :/

The kev_msg_post function is not documented, save for within a header file, kern_event.h:

Its only parameter is a kev_msg structure that is also presented in the same .h file:

The kev_msg structure is fairly simple. First, there are various members within the structure that classify the kernel event. As will be shown, this allows user-mode code to 'register' for, or filter on specific broadcast events. The first member, the 'vendor_code' should be populated with an integer that represents a custom vendor id (such as 'com.objective-see'). Use the kev_vendor_code_find function to 'lookup' a vendor code from an arbitrary string. The remaining classification members (kev_class, kev_subclass, and event_code), can be populated with values such as KEV_ANY_CLASS, KEV_ANY_SUBCLASS, and an abitrary integer for for the event code (I #def'd a custom constant, PROCESS_BEGAN_EVENT, to 0x1). Besides the members that deal with classifying the kernel event, the final member of the kev_msg structure is an array of five kev_d_vectors structures. These structures contain a length and pointer to event data and should be populated with the data to broadcast to user-mode. Once the kev_msg structure is filled out, the kev_msg_post function should be called to post the message to user-mode.

The following code illustrates the following paragraph, in a more digestible manner:

//kext's/objective-see's vendor id
u_int32_t objSeeVendorID = 0;

//start
// ->find & save vendor ID
kern_return_t start(kmod_info_t * ki, void *d)
{
   ...

   //find vendor ID
   kev_vendor_code_find("com.objective-see", &objSeeVendorID);

   ...
}

//kauth callback for KAUTH_FILEOP_EXEC events
// ->send info about process creation to user-mode via kev_msg_post()
static int processExec(kauth_cred_t credential, void* idata, kauth_action_t action, uintptr_t arg0, uintptr_t arg1, uintptr_t arg2, uintptr_t arg3)
{
   //kernel event message
   struct kev_msg kEventMsg = {0};
   ...

   //set vendor code
   kEventMsg.vendor_code = objSeeVendorID;

   //set class
   kEventMsg.kev_class = KEV_ANY_CLASS;

   //set subclass
   kEventMsg.kev_subclass = KEV_ANY_SUBCLASS;

   //set event code
   kEventMsg.event_code = PROCESS_BEGAN_EVENT;

   //add pid
   kEventMsg.dv[0].data_length = sizeof(pid_t);
   kEventMsg.dv[0].data_ptr = &pid;

   //add uid
   kEventMsg.dv[1].data_length = sizeof(uid_t);
   kEventMsg.dv[1].data_ptr = &uid;

   //add ppid
   kEventMsg.dv[2].data_length = sizeof(pid_t);
   kEventMsg.dv[2].data_ptr = &ppid;

   //add path
   kEventMsg.dv[3].data_length = (u_int32_t)strlen(path)+1;
   kEventMsg.dv[3].data_ptr = path;

   //broadcast msg to user-mode
   status = kev_msg_post(&kEventMsg);

The complete code for BlockBlock's kext, which fully implements both process creation detection (via the kAuth subsytem), and broadcasting such detections to user-mode, can be downloaded as an Xcode project

Before discussing how code in user-mode can access this broadcast data, it should be noted that by definition, "broadcast" implies that multiple clients may be able to access or receive this data. As such, it's likely unwise to use the kev_msg_post function for transferring any sensitive data to user-mode. (Mahalo to @DubiousMind for pointing this out!)

Receiving the Data in User-Mode
Via the kev_msg_post function, data from kernel-mode may easily be broadcast to user-mode. In order to receive this data, code in user-mode can make use of a kernel event socket. This is described both in Apple's "Network Kernel Extensions Programming Guide" and J Levin's invaluable Mac OS X and iOS Internals book (pp. 657-658).

In short, three simple steps are required:

Create a kernel event (system) socket
Set the filtering options
Receive the data

Let's look at each of these steps in more detail.

step one
First, the user-mode ('client') application should create a kernel event, or system, socket. This can be accomplished via the standard socket function, passing in PF_SYSTEM, SOCK_RAW, SYSPROTO_EVENT for the domain, type, and protocol:

//system socket
int systemSocket = -1;

//create system socket to receive kernel event data
systemSocket = socket(PF_SYSTEM, SOCK_RAW, SYSPROTO_EVENT);

step two
Then, filtering options should be set on the socket. This tells the system, "hey I'm only interested in kernel events from a specific vendor." In other words, the BlockBlock user-mode component can filter for only broadcasts from the BlockBlock kext.

Recall that the kext set a vendor id, a class (KEV_ANY_CLASS), and a subclass (KEV_ANY_SUBCLASS) in the kev_msg, that was broadcast to user-mode. The user-mode code should specify these same constraints (filters). As in kernel mode, the vendor id must be retrieved from a vendor string (e.g. 'com.objective-see'). The ioctl function with the SIOCGKEVVENDOR request and a populated kev_vendor_code structure, can get the vendor id:

//struct for vendor code
// ->set via call to ioctl/SIOCGKEVVENDOR
struct kev_vendor_code vendorCode = {0};

//set vendor name string
strncpy(vendorCode.vendor_string, "com.objective-see", KEV_VENDOR_CODE_MAX_STR_LEN);

//get vendor name -> vendor code mapping
// ->vendor id, saved in 'vendorCode' variable
ioctl(systemSocket, SIOCGKEVVENDOR, &vendorCode);

Once the vendor id has been retrieved, a kev_request structure can be populated with this id, class (KEV_ANY_CLASS), and subclass (KEV_ANY_SUBCLASS). Again, the ioctl function should be invoked, this time with the SIOCSKEVFILT request and the populated kev_request structure. This will 'set' the filters options on the system socket:

//struct for kernel request
// ->set filtering options
struct kev_request kevRequest = {0};

//init filtering options
// ->only interested in objective-see's events kevRequest.vendor_code = vendorCode.vendor_code;

//...any class
kevRequest.kev_class = KEV_ANY_CLASS;

//...any subclass
kevRequest.kev_subclass = KEV_ANY_SUBCLASS;

//tell kernel what we want to filter on
ioctl(systemSocket, SIOCSKEVFILT, &kevRequest);

step three
With the kernel event (system) socket created and initialized, one can simply call the recv function in a loop to start receiving the broadcast data (e.g. process creation notifications) from the kext:

//foreverz
// ->listen/parse process creation events from kext
while(YES)
{
   //ask the kext for process began events
   // ->will block until event is broadcast
   bytesReceived = recv(systemSocket, kextMsg, sizeof(kextMsg), 0);

   ...

This data (after some sanity checks, i.e. on the bytes received), should be type-cast to a kern_event_msg structure:

//struct for broadcast data from the kext
struct kern_event_msg *kernEventMsg = {0};

//type cast
// ->to access kev_event_msg header
kernEventMsg = (struct kern_event_msg*)kextMsg;

This structure contains useful fields such as the total size of the message, the event code (e.g. PROCESS_BEGAN_EVENT) and of course the data that was placed in the kev_d_vectors structure(s) by the kext.

Now the data (kern_event_msg structure), can be processed. First, you'll likely want to ignore any events from the vendor that are not one of interest. For example, the BlockBlock user-mode client application (currently), only cares about PROCESS_BEGAN_EVENT notifications:

//only care about 'process began' events
if(PROCESS_BEGAN_EVENT != kernEventMsg->event_code)
{
//skip
continue;
}

The data that was inserted into the message by the kext (before being broadcast via the kev_msg_post function), can be accessed via the event_data member of the kern_event_msg structure. Though in kernel-mode the data was inserted into various kev_d_vectors structures, in user-mode event_data points to a continuous buffer of the data.

Of course, the format of the received data will be kext specific. Recall that the BlockBlock kext inserts, then broadcasts the pid, uid, ppid, and path of newly created processes in each message. As such, the user-mode application can access and parse (and process) this data in the following manner:

//custom structure
// format of data that's broadcast from kext
struct processStartEvent
{
   //process pid
   // ->id's all chunks
   pid_t pid;

   //process uid
   uid_t uid;

   //process ppid
   pid_t ppid;

   //process path
   char path[0];
}

//typecast custom data
// ->begins right after header
procStartEvent = (struct processStartEvent*)&kernEventMsg->event_data[0];

//now pid, uid, etc can be accessed :)
syslog(LOG_DEBUG, "path: %s \n", procStartEvent->path]);
syslog(LOG_DEBUG, "pid: %d ppid: %d uid: %d\n", procStartEvent->pid, procStartEvent->ppid, procStartEvent->uid);

With both the kernel- and user-mode complete, time to see if this actually works. It does!

Limitations of the kev_msg_post Function
As the image above shows, things appear to be working well. The kext is broadcasting all new process creation notification events, which are then consumed by the user-mode application. However, after watching the logs for awhile, an error occasionally pops up:

Specifically, the kev_msg_post function occasionally fails with error 40. In errno.h, the error 40 is defined as EMSGSIZE, indicating that a message is too long:

#define EMSGSIZE 40 /* Message too long */

Consulting the kern_event.h header file that contains the function prototype for kev_msg_post function, Apple states "if the message is too large, EMSGSIZE will be returned." This is consistent to when the error is seen in the logs - that is to say, large messages (e.g. ones with looooonnng process names) trigger the error.

The solution is of course to simply broadcast shorter messages (chunking longer ones into several messages). But what is the maximum size of the message that kev_msg_post can handle? There is no documentation for this function, and the header file where the function is declared provides no information about this value - so off to google!

Unfortunately while others have articulated this exact question, nobody (AFAIK) provided a conclusive answer:

Now a real software developer would likely generate tests (i.e. with larger and larger messages) to uncover the maximum message size. However, this seems like a good opportunity for some kernel reverse-engineering -which IMHO, is far more interesting than writing tests ;)

The kev_msg_post function is implemented in the kernel proper (/System/Library/Kernels/kernel). Disassembling this function reveals the answer to our question almost immediately: 224! ...Take that test-driven-approach :P

The disassembly shows the kev_msg_post function calculating the total size of the kern_event_msg structure that is delivered to user-mode. Specifically, it starts with the size of the header, KEV_MSG_HEADER_SIZE (0x18), then adds the size of the data in each kev_d_vectors dv structure (kev_d_vectors dv[0].data_length -> kev_d_vectors dv[4].data_length). If this cumulative value (EAX) is not below 0xE1 (225 decimal), the kev_msg_post function returns with 0x28 (40 decimal), EMSGSIZE.

Knowing the maximum size of data that kev_msg_post can handle, the code in the kext can be updated to chunk data that would exceed this limit (see code below). Note that extra logic is needed since each message always contains the pid, uid, and ppid of the newly created process (to allow the user mode to reassemble the chunks). Also, since the maximum size (224), includes the size of the message header (KEV_MSG_HEADER_SIZE), this must be accounted for as well:

//chunk pointer
char* chunkPointer = NULL;

//start at path
// ->this works, since data before path is small enough to never chunk
chunkPointer = path;

//send all chunks
// ->termination; end of path
while(0x0 != *chunkPointer)
{
   //add current offset of path
   kEventMsg.dv[3].data_ptr = chunkPointer;

   //set size
   // ->either string length (with NULL)
   // or max size - pid, etc and extra for NULL!
   kEventMsg.dv[3].data_length =
            min(strlen(chunkPointer)+1, (MAX_MSG_SIZE-nonPathSize-1));

   //broadcast msg to user-mode
   status = kev_msg_post(&kEventMsg);

   //advance chunk pointer
   chunkPointer += kEventMsg.dv[3].data_length;
}

Of course the user-mode code must be updated as well to handle partial chunks and full re-assembly. However, this can be accomplished in a few lines of code.

Conclusions
Well this wraps up the three-part blog mini-series. To recap, part I showed how to monitor process creations via a MAC policy, while part II achieved the same goals with a KAuth listener. Finally, this post concluded by showing one way that the data from kernel mode could easily be transmitted to a user-mode application.

As always, shoot me an email if you have comments, suggestions, or any other feedback!