The Mis-leading 'Active' in PsActiveProcessHead and ActiveProcessLinks

On Windows, the nt!PsActiveProcessHead symbol points to a doubly-linked list of EPROCESS objects. It is very well documented as being the primary source of "current" or "active" processes. However, while using Volatility to analyze a Windows 7 memory dump, Gleeda and I discovered that the pslist command was showing something odd. In fact, I noticed the odd behavior a while back and never got a chance to look into it. The odd thing was that the pslist included info for some terminated processes. But shouldn't everything in PsActiveProcessHead and ActiveProcessLinks be, well...active?

For example, take a look at the following output. The MSBuild.exe and 4 copies of vcpkgsrv.exe are all in the doubly-linked list, but none of them are still running.

$ python vol.py -f Windows7.vmem --profile=Win7SP0x86 pslist
Volatile Systems Volatility Framework 1.4_rc1
Name Pid PPid Thds Hnds Time
System 4 0 94 467 2011-02-16 14:18:23
smss.exe 256 4 2 29 2011-02-16 14:18:23
csrss.exe 344 336 10 718 2011-02-16 14:18:33
[snip]
svchost.exe 3388 512 14 346 2011-02-16 14:20:58
devenv.exe 3640 1148 20 713 2011-02-16 14:21:29
MSBuild.exe 2732 3640 0 ------ 2011-02-16 14:30:01
vcpkgsrv.exe 3900 3640 0 ------ 2011-02-16 14:39:52
vcpkgsrv.exe 3496 3640 0 ------ 2011-02-16 14:52:28
vcpkgsrv.exe 3984 3640 0 ------ 2011-02-16 14:52:38
vcpkgsrv.exe 188 3640 0 ------ 2011-02-16 14:56:19
[snip]

To make sure the output is accurate, I cross-referenced the data with WinDbg on the machine from which the memory dump was acquired. Using a command presented in Chapter 14 of Malware Analyst's Cookbook (and later enhanced by Matthieu Suiche), I asked the debugger to print information on the objects in PsActiveProcessHead. Sure enough, it too shows that these inactive processes are still in the active process list:

kd> !list "-t nt!_EPROCESS.ActiveProcessLinks.Flink -e -x \"dt nt!_EPROCESS ImageFileName\"(poi(nt!PsActiveProcessHead)-@@c++(#FIELD_OFFSET(nt!_EPROCESS,ActiveProcessLinks)))"

[snip]

kd> dt nt!_EPROCESS ImageFileName 0xffffffff8435ba58
+0x16c ImageFileName : [15] "MSBuild.exe"

kd> dt nt!_EPROCESS ImageFileName 0xffffffff8436e548
+0x16c ImageFileName : [15] "vcpkgsrv.exe"

kd> dt nt!_EPROCESS ImageFileName 0xffffffff86177030
+0x16c ImageFileName : [15] "vcpkgsrv.exe"

kd> dt nt!_EPROCESS ImageFileName 0xffffffff843c0380
+0x16c ImageFileName : [15] "vcpkgsrv.exe"

kd> dt nt!_EPROCESS ImageFileName 0xffffffff854d91a8
+0x16c ImageFileName : [15] "vcpkgsrv.exe"

[snip]

How do I know that these processes are not actually running? Well for one, Volatility reports zero threads and an invalid EPROCESS.ObjectTable (handle table) pointer. Plus tools like SysInternals Process Explorer don't show anything about them.

So the questions are:

1) When do processes get removed from the PsActiveProcessHead
2) What are the conditions (if any) that must be met before removal takes place

Until recently, I thought that the answer to number 1 was obvious - a process is removed from PsActiveProcessHead immediately after it terminates. But that's not the case. I searched a bit in Windows Internals 5th Edition, which discusses process creation in great detail. However, it doesn't cover too much of process termination. Instead I used the trusty IDA Pro to find which function is responsible for removing a process. It happens inside nt!PspProcessDelete:

Now the question changes to "when is PspProcessDelete called." It isn't called directly by any function. Rather, it is moved into the DeleteProcedure member of an OBJECT_TYPE_INITIALIZER structure when the kernel first defines the default behaviors for all EPROCESS objects. This happens in nt!PspInitPhase0:

After seeing these clues, the life cycle of an object came into mind. Also as described in Windows Internals, the Object Manager tracks handles and references to an object, and only allows the object to be freed when the reference count reaches zero. And now it makes sense how a process can terminate but stay in the PsActiveProcessHead linked list. Here's a description of what I believe to be the most likely explanation:

1) Process A (the parent) creates Process B with an API such as CreateProcess. The PROCESS_INFORMATION parameter to this API is filled with a handle to Process B if the creation is successful.

2) Process B is terminated, either on its own or as the result of a user killing it. At this point, the thread count reaches zero, the EPROCESS.ObjectTable is set to NULL, and the EPROCESS.ExitTime is filled in.

3) Process A fails to call CloseHandle to decrement the reference count on Process B. Thus, the DeleteProcedure for Process B is never called and it is never removed from PsActiveProcessHead.

To test out my theory, I wrote a simple program that looks like this:

void main (void)
{
STARTUPINFOA StartupInfo;
PROCESS_INFORMATION ProcessInfo;

ZeroMemory(&StartupInfo, sizeof(STARTUPINFOA));
ZeroMemory(&ProcessInfo, sizeof(PROCESS_INFORMATION));
StartupInfo.cb = sizeof(STARTUPINFOA);

CreateProcessA(NULL, "notepad.exe", NULL, NULL, FALSE, 0, NULL, NULL, &StartupInfo, &ProcessInfo);

getchar(); //block until user presses a key

CloseHandle(ProcessInfo.hThread);
CloseHandle(ProcessInfo.hProcess);
}

The program creates a child process (notepad.exe) and remains running. This gives you a chance to use tools and verify the two new processes were created. Then you can close notepad.exe and watch it disappear from Process Explorer, but remain in PsActiveProcessHead (via the WinDbg command or Volatility's pslist). It is only when you type a character into my sample program that it progresses past the getchar() call. At this point, it closes its handle to notepad.exe, the reference count reaches zero, the EPROCESS object is marked for deletion, and it is removed from PsActiveProcessHead.

Before today, I would have said that an object scanning technique was the only way to identify terminated processes in a memory dump, but now we have another. Of course, it takes some getting lucky - for example a malware sample that launches various child processes and never closes the handles even after the child processes die.

And that, my friends, is the story of the mis-leading 'Active' in PsActiveProcessHead and ActiveProcessLinks!