NOTE: While this article focuses on ASMX services, the same server-side performance tuning principles can be applied to WCF services as well.  

 

Typical Web Service Configuration

A typical services hosting configuration involves a set of ASP.NET services (ASMX services) or WCF services hosted inside an HTTP Server (IIS Server).  These services (ASMX or WCF) access data repositories (either relational databases or XML data sources) through a Data Access Layer (and optionally an ORM layer).

 

Desktop client(s) and web clients connect to the hosted services and invoke one or more services over HTTP (perform service operations).

 

What can go wrong?

In the above ‘typical’ configuration, there are several hops between a client and the data that it needs to retrieve.

Hop 1 – From the Client (browser or desktop client) to the ASMX service.

Hop 2 – The ‘tiers’ in the service – assuming a regular 3-tier application – the hop from the ASMX to the business layer.

Hop 3 – The hop from the business layer to the data access layer.

Hop 4 – The hop from the data access layer to the database.

 

Each of these ‘hops’ can be a potential performance bottleneck. In this article, we will focus on the two most important hops – from the client to the hosted service – and from the Service to the Database (either via a DataAccessLayer or a persistence layer or a combination of the two).

Recently, I had the opportunity to troubleshoot in an environment where one or more of these hops were problematic. Since the system was already in production, time was of the essence. Essentially, instead of tackling it on a ‘hop by hop’ basis, I decided to break it up into ‘things that can be tried quickly (quick fixes)’ versus ‘things that take a little bit of time to try out (not so quick fixes – involving code changes)’.

The ‘quick fixes’ detailed here require no code changes. The only files that are affected are configuration files. The entire codebase, apart from the .config files, remains unchanged. The ‘not-so-quick’ fixes involve code changes – some fairly consuming code refactoring(s) etc.

 

Things to try first (‘Quick Fixes’ – no code changes required)

Since our system was already in production, time was of the essence. I decided to try a few 'quick fixes’ which would not take more than half a day to try out in a development environment. While these ARE important changes and CAN make your application perform better, these are in no way a substitute for the ‘Not-so-quick’ fixes which involve code clean up. While the author does not guarantee that any of these quick fixes will address your specific problem, the author DOES guarantee that the ‘Not-so-quick’ fixes will improve the performance of your Web Service and Client applications. Ideally, the two sets of changes (quick and not-so-quick), need to be done in parallel – for best performance results.

Some ‘quick’ fixes we will discuss in this post include;

1. Quick Fix 1 (For ASP.NET 1.1 only) – Part 1: Configuration parameter - Client Side maxConnections
2. Quick Fix 1 Part 2 (For ASP.NET 1.1 applications only): Configuration parameters - Server Side maxConnections, MinFreeThreads and MaxWorkerThreads
3. Quick Fix 2 – Part 1: Upgrade to IIS 7.0 and Windows Server 2008
4. Quick Fix  2 – Part 2: Performance tuning IIS 7 .0 to work with ASP.NET
5. Quick Fix 3 – Create individual Application Pools (IIS 6.0 upwards)

 

Quick Fix 1 (ASP.NET 1.1 and earlier only): The maxConnections Configuration parameter

 Note that if your ASP.NET’s runtime is version 2.0 or above, you can skip Quick Fix 1 altogether and move to Quick Fix 2.

 

With 2.0 and above, there is a machine.config setting that takes care of handling all the incoming HTTP requests (and handles more than 2). This setting is the autoconfig = true in the machine.config’s processModel element.

<system.web>
    <processModel autoConfig="true" />
</system.web>

 

Several configuration parameters (in both web.config and machine.config) control the performance of an ASP.NET web application. The most commonly known and tweaked parameter is possibly the ‘maxConnections’ which limits the number of both inbound and outbound HTTP connections.

 

This parameter affects the performance on both the server as the client – and needs to be tweaked on both sides. However, the server configuration needs to be tweaked in conjunction with additional parameters (Thread Pool configuration) – in order to optimize the number of requests handled on the server. If we increase the number of http requests coming into the server, we need to ensure that the server is equipped to handle multiple incoming requests. This is done by ensuring that the server has enough available threads to process the incoming requests in parallel. The server side configuration changes to the thread-pool parameters is also described below.

 

Quick Fix 1 (Pre ASP.NET 2.0 versions only ): – Part 1: Client Side maxConnections

Note that if your ASP.NET’s runtime is version 2.0 or above, you can skip Quick Fix 1 altogether and move to Quick Fix 2. 

On the client desktop running the desktop app, the app.config file is the one of interest. This contains the default maxConnections = “2”. We need to change this to a higher limit.

A good number to try for your outbound maxConnections (as per MSDN documentation) is 12.

DESKTOP - app.config 

<system.net>
    <connectionManagement>
      <add address ="*" maxconnection="12"/>
    </connectionManagement>
</system.net>

 

Quick Fix 1 (ASP.NET 1.1 only):  – Part 2: Server Side maxConnections, MinFreeThreads and MaxWorkerThreads

On the machine running IIS and the web service, both the web.config (for the application containing the web service) and the machine.config files need to be tweaked. The machine.config needs to be changed because the threading parameters that are in the <processModel> element need to be changed at the machine level.
Web.Config (for the ASP.NET application containing your asmx service)

<system.net>
    <connectionManagement>
      <add address ="*" maxconnection="12"/>
    </connectionManagement>
</system.net>

 

Machine.config (on the machine running IIS and hosting your webservice, )

The machine.config contains settings for the thread pool. The reason we need to tweak the thread pool on the server is that if we increase the number of requests that the server can handle, we have to ensure that the number of threads available are sufficient to handle the multiple requests.

The two parameters that are available for tweaking the thread pool are maxWorkerThreads and minFreeThreads.

The basic equation to be obeyed is: maxConnections = maxWorkerThreads – minFreeThreads

The idea is that if you have 12 maxConnections, that means 12 worker threads should be available for processing incoming HTTP requests. On the other hand, there may be several other task requiring available threads (background workers, callbacks etc.). To handle these, the minFreeThreads needs to high enough that these tasks do not interfere with the processing of the incoming requests (i.e. – there is no vying for the request-processing threads).

 

  To make both these ‘thread pool contenders’ happy (the incoming http requests as well as miscellaneous background tasks), the maxWorkerThreads and minFreeThreads need to be arranged so that their difference is the number of ‘somewhat guaranteed’ request-processing threads (‘somewhat’ because if the number of background tasks exceeds 88 then they will start looking at our 12 request processing threads for an available thread. However, setting minFreeThreads high enough – 88 – should ensure that this doesn’t happen).

 

The minFreeThreads is how many threads will be available for miscellaneous background processing tasks.  In our example above (setting the maxConnections to the recommended 12), maxWorkerThreads can be set to 100  and minFreeThreads set to 88. This allows for 12 threads to run simultaneous aspx requests and leaves 88 threads are available for miscellaneous background tasks (such as callbacks, background workers etc.) 88 threads should be high enough that the background tasks do not run out of threads and cast a hungry look at the 12 remaining threads (that we were dedicating to request processing). The snippet below includes all the changes that need to be made on the server side.

 

COMPLETE SERVER SIDE CONFIGURATION (ASP.NET 1.1 only):

(Note: do not remove or change the autoConfig=”true”, leave it intact).

 

Machine.config (applies to all applications using the thread pool)

<system.web>
   ...
   ...
   <httpRuntime minFreeThreads="88" />
   <processModel maxWorkerThreads="100" maxIoThreads ="100" autoConfig ="true"/>
</system.web>

 

Web.config (per individual application)

<system.net>
    <connectionManagement>
      <add address ="*" maxconnection="12"/>
    </connectionManagement>
</system.net>

 

NOTE: If both the client and the server are on the same machine (e.g. possibly in your development environment), then just making the changes in the machine.config may be enough – since that will make the change machine wide and across multiple applications. In case you do not want the maxConnections changed for certain hosted applications, you can always override the machine.config settings by changing it in the specific web.config.

 

Quick Fix 2 – Part 1: Upgrade to IIS 7.0 and Windows Server 2008

IIS 7 has a number of performance improvements that were not part of IIS 6. These features are discussed in detail in another blog post. To utilize these features, the server needs to be upgraded to Windows Server 2008 as well. If you need assistance with the upgrade, there are some good resources online – including this one. Once the upgrade to IIS 7 and to Windows Server 2008 is complete, certain web.config parameters need to be added/modified to make optimal use of IIS 7’s integration with ASP.NET. These parameters are discussed in detail below.

 

Quick Fix 2 – Part 2 Performance Tune ASP.NET to work with IIS 7

We will look at some of the easy, yet powerful possibilities in the web.config file. By taking advantage of these few tricks we can increase the performance of any new or existing website without changing anything but the web.config file.  The following XML snippets must be placed in the <system.webServer> section of the web.config.

HTTP compression

You’ve always been able to perform HTTP compression in ASP.NET by using third-party libraries or own custom built ones. With IIS 7 you can now throw that away and utilize the build-in compression available from the web.config. Add the following line to enable HTTP compression:

<urlCompression doDynamicCompression="true" doStaticCompression="true" dynamicCompressionBeforeCache="true"/>

By default, only text based content types are compressed.

doDynamicCompression

Setting this attribute to true enables compression of dynamically generated content such as pages, views, handlers. There really aren’t any reasons not to enable this.

doStaticCompression

This attribute allows you to decide whether or not you want static files such as stylesheets and script files to be compressed. Images and other non-text content types will not be compressed by default. This is also something you want to enable.

dynamicCompressionBeforeCache

If you do output caching from within your ASP.NET website, you can tell IIS 7 to compress the output before putting it into cache. Only if you do some custom output caching you might run into issues with setting this to true. Try it and test it. If your website works with this enabled, then you definitely want to keep it enabled.

Tip

By default, only text based content types are compressed. That means if you send application/x-javascript as content type, you should change it to text/javascript. If you use some custom modules in your website, then you might experience conflicts with the IIS 7 compression feature.

 

Quick Fix 3: Create application pools (IIS 6 upwards) for each hosted website.

 

A poorly performing website will typically affect another website if they share the same application pool (default). It is best to isolate each application in its own application pool. This is a simple change in IIS – the only downside is that IIS needs to be restarted in order for this change to take effect.

 

Summary of Quick Fixes

We explored tweaking the maxConnections (for pre-.NET 2.0 applications) on both the client and the server. The server side maxConnections has to be done in conjunction with a couple of other thread-pool configuration parameters – minFreeThreads and maxWorkerThreads.

 

  We also looked at upgrading to IIS 7 to utilize all of its performance improvements. We looked at the important web.config changes that makes use of new IIS 7 features so it integrates tightly with our ASP.NET application. In addition, we also looked at a simple IIS change which involves allocating individual pools (application pools) for each IIS application. 

In the author’s opinion, these 3 fixes are less time consuming than the ones detailed below – especially if one has a robust development environment to try them out.

 

Resources

• Add new mime-types for compression
• Configure HTTP compression in IIS 7 (Technet)
• MSDN Knowledge Base – maxConnections
• Upgrading to Windows Server 2008 (from Server 2003)

Things to try second  (involves code changes)

Once you have exhausted the list of ‘quick fixes’, you are ready to tackle some of the important code changes. At the cost of being repetitive, the author would like to encourage you to allocate time for these changes. These changes, in the author’s experience, will almost guarantee various levels of performance improvements. The ‘quick-fixes’ may amount to little if the server code is poorly written.  Depending on how much of your code requires to be modified, these can take anywhere from a few days to several weeks. The code modifications that we will discuss here include:

 

Client Side Code Modifications

1. Switch to asynchronous client side invocations
2. Correctly dispose all client side resources

Server Side Code Improvements

1. Write a Custom HTTP handler to process incoming asynchronous calls
2. Improve the performance of your data access layer
3. Perform Custom Authentication
4. Refactor large web service classes

Client Side Code Modification 1: Asynchronous client side service invocations 

 

Motivation: One of the drawbacks of calling services synchronously is the possibility of ‘locking’ your user interface. If a service takes a while to process, the client  may stay unresponsive to user events etc. while the request is being processed by the server. To ensure that this does not happen, ensure that all service calls are made asynchronously.

Caveat:  While the idea of changing every service call to Asynchronous sounds like a clear winner, there are a lot of cases where you would not want to do this. Take the simple example of two successive service calls – say 'PlaceOrder’ and ‘SendInvoiceToCustomer’. So – a customer places an order on your website (using the ‘PlaceOrder’ service) and once the order is placed successfully, you invoice the customer (using the SendInvoiceToCustomer’ service). In a synchronous world, this would be easy to program – you would simply call PlaceOrder first  - wait till it was successful – and then call SendInvoiceToCustomer.

In an asynchronous world, things are different. Since you basically call both of these methods asynchronously, you have no idea when they will actually complete. So – the ‘SendInvoiceToCustomer’ could actually return before the ‘PlaceOrder’ – which means you will invoice a customer for an order that has not yet been placed! Bad!

There are ways around this in an asynchronous world – but the point is that we need to carefully address our workflows when we are moving from a synchronous world to an asynchronous world. There are a few other ramifications such as ExceptionHandling which would work differently in an asynchronous world as well.

Assuming that we do not have complex dependencies in our workflows (such as outlined above), we can move ahead with changing our synchronous service calls to asynchronous.

The example below shows how to call a webmethod named ‘GetCustomerSalesInformation’ asynchronously and having the results automatically populate a UI element (a Textbox in this example). An intermediate step in the HandleCallback involves getting a handle to the AsyncState object – which is used to call the ‘EndCustomerSalesInformation’ method.

 

Client Side Code Modification 2:  Correct Disposal of client side resources

Both Finalize and Dispose try to do the same thing – dispose of unused resources. The problem arises because Finalize is non-deterministic (we do not know when it will be called). On the other hand, all clients need to ensure that they call Dipose() on the object they are using once they are done using it. If a client calls Dispose() and it turns out that the GC had already ‘Finalized’ this object – there will be no object left to Dispose. This results in an exception The IDisposable pattern makes it possible to ensure that client side resources are disposed correctly.

Correct usage of Dispose() and Finalize() for the Business Layer of your application

Code Snippet


public interface IDisposable
{
  void Dispose();
}
 
public class BusinessObject: IDisposable
{
   public void Dispose()
   {
       // do cleanup
       // ....
 
       // call Base.Dispose()
       base.Dispose();
   } 
}

If all your objects implement this interface, then it makes life easier for the client. All the client needs to do is check whether the object has implemented IDisposable. It can do this as follows:

Code Snippet


class Client 
{
    // If all your objects implement this interface, then it makes life easier for the client. All the client needs to do is check whether the object has implemented IDisposable. It can do this as follows:
     
    // Check – prior to calling Dispose() on the object – whether the object has implemented IDisposable
    public void SomeMethod()
    {
        IDisposable dispObj = obj as IDisposable;
        if (dispObj != null)
        {
            dispObj.Dispose();
        }
    }
}

Server Side Code Modification 1:  Implement Asynchonous HttpHandler

While ensuring that your client calls the services asynchronously is a best practice, the client calls are only part of the puzzle. To make sure that the asynchronous calls are received and processed in an optimal manner, the server needs to be equipped to handle multiple incoming asynchronous requests. As it so happens, ASP.NET provides us with just such a capability in the form of custom HTTPHandlers. A custom HTTP handler can intercept an HTTP request at any point in the request processing pipeline. 

The key elements of implementing the IHttpAsyncHandler are shown in the code snippet below – for a sample web service called GetCustomerInfo. There are two other small steps after implementing the code below – adding the new HttpHandler to the web.config file  - and register an IIS application mapping for the newly created extension type (specified in the web.config). These two steps are detailed below as well.

 

Step 1: Implement IHttpAsyncHandler

Step 2: Add our new AsynchHandler to the web.config file

Step 3:  In IIS, register an ‘application extension’ mapping for the path defined in the web.config above (*.CustomerAsync). This is done easily through IIS Manager using these instructions.

Summary of Server Side Code Modification 1:

This approach (of implementing a custom HttpHandler to handle Asynchronous calls) should show a marked improvement in the performance of the sample service ( GetCustomerInfo in our example). As you can see, this approach has a drawback in that it needs to be done on a service-by-service basis. If your application has an obvious suspect service that is performing poorly, it would be an ideal one to write an HttpAsyncHandler for.

 

Server Side Code Modification 2: Faster, Cleaner Data Access Layer

 

 An Introduction to ADO.NET  provides a good overview of ADO.NET, with v1.1 features as well. Furthermore, although this section is named Data Access, it could also refer to XML sources as a viable data sources. Therefore, Reading, Storing and Transforming XML Data in .NET is also recommended.

 

  This ‘cleanup’ involves designing efficient database design and queries as well as tuning ADO.NET techniques for faster retrieval of data.

Efficient database design and queries

• Normalize your data for maximum efficiency and faster performance.
• When creating new SQL tables always consider the best data types to employ with the type of column data you'll be storing.
• Index your tables using SQL's Create Index command and or Query Analyzer's Index Tuning Wizard, assuring that your databases are finely tuned.
• Use SQL Stored Procedures for all your data access, as SQL Server compiles them for all future uses. Furthermore, in SQL, straight and narrow is the best method in writing SQL Stored Procedures. Never overcomplicate your procedures with unnecessary or excessive temporary tables, server-side cursors , as these do create performance bottlenecks. You're better off creating subqueries off a base query and or with a join (avoiding left joins).
• Never name your stored procedures with a "sp_" prefix, as SQL will interpret this as a system procedure.
• Create SQL Views for added clarity and security.
• Include SET NOCOUNT ON in all your stored procedures, as this diminishes network traffic by eliminating the need for SQL in always returning how many rows the procedure affected.
• Use sp_executesql to execute any standard non-Sproc SQL statements in your code, gaining Stored Procedure-type benefits and reducing overhead:

  sp_executesql N'Select * from table 

  and even within a Stored Procedure, instead of strictly EXECing any query strings:

  DECLARE @sqlQuery nvarchar (100) 
  SET @sqlQuery = N'SELECT * FROM database.dbo.table' 
  EXECUTE sp_executesql @sqlQuery

• Finally, optimize SQL Server itself to maximize performance. Read SQL Server Settings Optimization Tips for more info.
  ADO.NET
• Create all your database access routines as generic, versatile objects, rather than client-side repeated-code methods. All your UI should do is interact with these components, and not have to work out any details. The methods aforementioned in the Presentation Layer apply here as well in creating components and controls.
• DataReaders versus DataSets: If you want to page data or provide your application with functionality, use a Dataset as the preferred method of disconnected data, otherwise use a Datareader for all your data retrieval. For XML users this would translate to an XmlReader , and StreamReader for text files, both equivalent as that of a DataReader for their respective file types.
• Use the correct managed data provider for your particular database, ex. System.Data.SqlClient for SQL Server , System.Data.OleDb for Access, System.Data.OracleClient for Oracle , etc.
• Use Strongly-Typed Datasets over the standard, common un-Typed ones when possible, as this yields better performance. A generated typed Dataset is basically an early bound class inheriting the Dataset base class, as opposed to the typical late bound, non-inheritable Dataset. Thus, it allows for greater flexibility in dealing with your data. In turn, you'll be dealing with easier to read code, whereby you can access fields and tables by customizable names, instead of the conventional collection-based way. A good introduction to Using Typed DataSets can be found here.
• Utilize .NET Caching - as this will significantly boost performance and greatly diminish any persistent database interaction. Howeverf you decide on caching your data from within your data object, then conventional caching methods won't apply. Within the confines of components, and its interaction with the caller page, these requests happen via an HTTP request. Therefore, caching within your component can only be implemented by using the HttpContext.Cache Class property, part of .NET's Page class.
• Use parameterized Stored Procedures alongside .NET's Command Class Prepare() method, that caches your query for all future SQL Server uses.

  objCommand.Prepare()

• Make chunky calls to your database rather than smaller, chatty calls. It's better to group all similar, associated calls in one SQL Server access. In other words, use one solid connection to retrieve as much as you can, as opposed to multiple ones.
• Avoid using Universal Data Link ( UDL ) files for OleDb connections as these can cause potential performance hits. .NET's SQLClient managed data provider does not support this, as prior versions of SQL Server did.
• Take advantage of connection pooling (whereby all your connection strings are identical) by storing all your connection strings in your web.config file. Furthermore, you'll find that your application will scale a lot better with any increased network traffic by doubling, even tripling the default Max Pool Size of 100 to 300, and even bumping up the default Connect Timeout of 10 seconds to 60.
• Additionally, if your not enlisting any transactional procedures, include enlist=false ; to the database connection string(s) for added performance.
  Example:

  <configuration>
    <appSettings>
      <add key="myDatabase" value="User ID=id; Password=pw; Data Source=datasrc;Initial Catalog=dbCat; Enlist=false;
           Connect Timeout=60;Min Pool Size=10; Max Pool Size=300;"/>
    </appSettings>
    <configuration> 
  and call it from your page, code-behind source file or component like so:          

  [C#]             
  ConfigurationSettings.AppSettings["myDatabase"].ToString();

• Finally, remember to close, clear and dispose of all your data objects no matter what. If you would like to further confirm that your database connection is indeed closed, you would write:

  if (dbConnection.State != ConnectionState.Closed) { dbConnection.Close(); }

Server Side Code Modification 3: Perform Custom Authentication

 

Change from Integrated Windows to SQLServer based authentication. The performance overhead with using Integrated Windows authentication is high – at the cost of providing the highest possible security. However, using a combination of a simple username/password (stored in a trusted database), one can gain signficance performance out of each service call which uses the SQL Server authentication. An example and some code samples are forthcoming – please check back here soon.

 

Server Side Code Modification 4:   Refactoring large Service Classes

All too often, when I encounter a new webservice project, I notice a huge, single monolithic ASMX class that contains ALL the services that  need to be exposed. This is usually done to ensure that the client(s) avoid adding multiple web-references and only need to add a single webreference. While this is a valid client-side concern, this is a somewhat avoidable memory hog situation on the server. Memory Hog because every single time a single webmethod is invoked by the client – the entire giant ASMX class is instantiated – just so a single method can be invoked. Do this a few times – and you may have several dozen instances of a large class sitting in memory – waiting to be garbage collected. Not an ideal situation.

 

There’s a couple of options here. The standard, recommended refactoring option is to simply break up the single webservice class into multiple, smaller classes.  Each smaller webservice class would cater only to its own business entity – e.g.  CustomerService (for Customer related service calls), AdminService (for admin related service calls) etc.

 

What about the client? Does each client now have to add multiple webreferences? Not if they don’t want to. Inspite of having multiple services hosted within IIS, the client can still connect to just a single service endpoint – since the multiple services can be combined into a single endpoint. This can be accomplished using a manual tool called wsdl.exe

 

We need to  generate the wsdl using the wsdl.exe tool provided with the IDE (instead of letting the Proxy Generator generate it automatically).

 

The manual command for doing this (using the above example of three different services) is:

Summary of Fixes Involving Code Changes (‘Not-So-Quick-Fixes)

We investigated performance issues in a typical Web Service scenario – with a desktop client (or a browser client) accessing IIS hosted web services. While this article focused on ASMX services, the same server-side performance tuning principles can be applied to WCF services as well. We assumed that the Web Service was already in production which is why we broke up our troubleshooting into ‘quick fixes’ and ‘not-so-quick’ resolutions. The ‘quick fixes’ included configuration changes, upgrades to IIS7 and Windows Server 2008 and some other tasks that were not super time consuming.

In no way is it implied that any of the quick fixes will provide guaranteed, immediate relief to a given Web Service’s performance.

The quick fixes discussed here are not meant to overcome poorly designed services or poorly written server side code. They merely utilize the existing resources (such as the IIS server’s integration with ASP.NET) much more fully.

However, the server side code fixes (not-so-quick fixes) discussed here – including refactoring, writing a custom Async HttpHander, writing a souped up Data Access Layer – GREATLY INCREASES the chances of eliminating performance bottlenecks in your production web services – whether they be ASMX services or WCF (http binding) services. For a complete discussion on WCF services and how they compare to ASP.NET ASMX web services the author would recommend another blog post of his.

In addition to the server side cleanup, we also discussed coding practices to ensure that the client does not encounter any memory leaks or sluggish UI responsiveness.

There are certainly many more areas that deserve investigation. These include:

• Performance of the persistence layer, if your application is using one
• Correct partitioning of larger ASMX classes
• UI best practices (other than the few discussed in this article)

. The author would welcome any real world experience that helped your web service(s) perform better.