Is there a way to expand the current WorkingSet of a process to 1GB? [duplicate]

To achieve what you want you need to call/pinvoke SetWorkingSetSizeEx with 1 GB as minimum (second param) and QUOTA_LIMITS_HARDWS_MIN_ENABLE as fourth param which makes sure that workingset size won't go below minimum you gave even in "high memory pressure" conditions of the system.

The system behaviour also depends on privileges of the caller, depending on OS version etc. you might need SE_INC_WORKING_SET_NAME and/or SE_INC_BASE_PRIORITY_NAME !

Another (nicer) option which uses these APIs "behind the scenes" is a .NET wrapper you can find here.

• I was going to post this, but it's not supported on XP and older. - Gabe
• @Gabe We are using Windows Server 2008 R2, so we don't need to support XP. - Contango
• @Yahia There is some nice code that shows how to use privileges, at processprivileges.codeplex.com. I'll try the code out this weekend. - Contango
• @Yahia I think you're right. The code at gate.upm.ro/os/Laborator/Utilitare_sistem_Windows/testlimit.c demonstrates that there must be a call to EnablePrivilege( SE_INC_BASE_PRIORITY_NAME ); before the call to SetWorkingSetSizeEx. Its in C, not C#, but I think it demonstrates the concept. - Contango

If your problem was that your process was getting its WS trimmed too agressively in low-memory situations, you could take care of it by calling SetProcessWorkingSetSize or just setting Process.CurrentProcess.MinWorkingSet.

What you've shown, though, is that your working set is getting reduced by a GC. This tells me that what's really happening is the GC is deallocating the pages that make up your WS. If this is the case, you have an address space problem rather than a working set problem, and there's no system call you can make to prevent it. Ideally you would be able to tell the GC not to return its memory to the OS, but .NET doesn't have a feature like that.

To solve an address space problem you will have to reuse objects you have already allocated. If your problem is with the Large Object Heap, it's most likely due to collections. For example, rather than creating a new array/list/dictionary, call its Clear method and reuse it. If your problem is strings, you might be able to get away with using StringBuilders to stay out of the LOH.

If you have certain types of object you create lots of, consider creating a pool of them that get recycled. I've never done such a thing, but if I were to implement it I would create an object with a static factory method that pulls objects out of a pool and calls an initializer instead of having public constructors, and put a finalizer on it that puts it back in the pool and nulls out any references in it. Depending on the needs, the pool might be a ConcurrentBag<WeakReference<T>>.

• Unfortunately, when GC.Collect() runs, it resets the workingset down to something tiny, e.g. 50MB. Is there a way to permanently set the working set to 1GB? - Contango
• @Gravitas this is a very interesting question, please let me know if you get this resolved - l--''''''---------''''''''''''
• @Gravitas to avoid that you need to call SetWorkingSetSizeEx (see my answer for details)... - Yahia
• @Yahia: Are you sure that it wasn't just because Gravitas isn't running his process with SE_INC_BASE_PRIORITY_NAME enabled? - Gabe
• @Yahia I tried using SetProcessWorkingSizeEx with QUOTA_LIMITS_HARDWS_MIN_ENABLE, but when GC.Collect() is called, it trims the working set anyway. - Contango

I think assigning the process to a Job Object and setting JOB_OBJECT_LIMIT_WORKINGSET and MinimumWorkingSetSize might work.

• Excellent comment, I'll try this on Monday. - Contango

The only way that we could find to increase the WorkingSet of a process under .NET, to reduce soft page faults, was to run the entire .NET application under a custom CLR Host. This is a non-trivial exercise, requiring about 800 lines of custom written, rather dense C++ code. The C++ code intercepts the .NET calls to the Win32 memory management methods, altering the behavior of the .NET runtime so it doesn't free memory as aggressively as it normally would.

This has the effect of incurring all of the soft page faults when the application starts up, so that during normal application execution, the number of soft page faults in the .NET app drops pretty much to zero.

This means that the application may be memory hungry, but it runs faster. In other words, we are sacrificing memory usage for increased realtime performance.

I think there is a misunderstanding here: The GC freeing unused memory is good. It is of no use to keep it around just to see a higher number in the WS metric... You don't want unused memory in the WS. You are optimizing for the wrong metric.

The number of page faults is not that meaningful because there are soft and hard faults. Hard faults read from disk. Soft faults mean nothing in most cases. They are an implementation detail of the OS. For example, for every newly allocated page you touch a soft fault occurs.

Here is some evidence that your red graph is not showing hard faults: At the end it shows about 500/sec. Your disk cannot deliver 500 IOPs so it cannot be hard faults.

You probably want to measure (and reduce) hard faults only. Only hard faults impact performance in a meaningful way.

Also, setting WS limits does not influence what the GC does. The GC is not an operating system component. Is is a user-mode library. It has nothing to do with what the OS decides to use as the working set. For that reason you cannot make the GC not release memory by setting WS limits.

So why does the WS shrink when the GC runs? Because the GC deletes some memory. Setting some WS limit cannot prevent deletion of memory.

In the comments it was suggested the GC might itself call some API to shrink the working set. I cannot see any reason at all why it would do that. Why would the GC force pages out of the process? Remember: WS and allocated pages are not the same (at all!). The GC might release memory. It does not force the WS to shrink. What purpose would that have?

• I also thought the OP should just ignore soft faults, but then I saw he posted "every time a page fault occurs, it blocks the thread for 250us". Would you prefer the question was "How to eliminate 250us delays after GC"? - Gabe
• This is a different question. I'd be interested to know in how this 250us delay way measured and how it was traced to soft faults. I cannot imagine that the soft fault performance hit is more than the GC performance hit. - usr
• @usr We measured the 250us delay using Microsoft Concurrency Visualizer, available in Visual Studio 2012. We could see a soft page fault occurring, which blocked other threads in the system from the running. - Contango