W3WP And SOM Processes Consume Too Much Memory When ArcGIS Server Services Are Under Heavy Load

On a server where memory is limited you might have noticed that the IIS worker processes (W3WP) and the ArcGIS Server SOM process may begin to adversely affect other applications by consuming large amount of memory when the ArcGIS Server services are under heavy load.

ESRI has provided an explanation and workaround for this problem. The reason that this occurs:

The .NET garbage collection has a concept of generations (Garbage Collector Basics and Performance Hints). Local variables and short term objects are classified as generation one and are cleaned up frequently when garbage collection is called. Objects that have longer lifetimes are classified as generation two or higher and are less frequently cleaned up when garbage collection is called.

ArcGIS Server DCOM runtime callable wrapper (RCW) proxies and their references to DCOM objects in the SOM process are particularly susceptible to being pushed into higher generations and accumulate on the Web server and SOM until the less frequent second or third generation garbage collection algorithm occurs.

Under load, when second and third generation garbage collection may be delayed due to its high CPU utilization cost, the W3WP and SOM process memory utilization can grow rapidly.

The workaround for this is quite simple. It’s a matter of decreasing the the .NET Garbage Collection (GC) internal to force the recovery of resources. This is just a one line change to the web.config for the services.

If memory is a critically limiting resource on the Web services machine, it may be desirable to force recovery of resources by decreasing the default .NET Garbage Collection (GC) interval. Decreasing the GC interval encourages the GC algorithms to clear out generation two and three objects no longer being used.

To address this issue, the ArcGIS Server Web services use a web.config setting to force garbage collection after a specified number of requests.

The default GC Interval (every 10,000 requests) has been chosen to address the memory consumption problem described above based on testing representative services under load.

If a problem with memory growth in the W3WP worker process or SOM process persists, the GC interval can be lowered from 10,000 to a smaller number (for example: 1000). This results in more frequent garbage collection. However, garbage collection is a CPU intensive operation and decreasing the GC interval results in the reduction of throughput. If a more frequent GC interval than 10,000 is chosen, select a number cautiously so that the throughput is not significantly affected.

At the moment it looks like this only applies to ArcGIS Server 9.2 services. I have yet to experience this problem in ArcGIS Server 9.3.

Instructions on how to go about adding this interval into the web.config and making this change can be found on the ESRI Support Centre here: http://support.esri.com/index.cfm…