Scalability Challenges with Web Services - ASP.NET

Ajax web sites tend to be very chatty because they make frequent calls to web services,e.g.,auto complete on text boxes,client-side paging on data grids, and clientside validations require frequent web service calls.Thus,Ajax web sites produce more ASP.NET requests than similar non-Ajax web sites. Moreover, it gets worse when the web service calls make another web service call to external services

Then you not only have an in coming request but also an out going request This means double the load on ASP.NET.ASP.NET has a limited number of worker threads that serve the requests.When there’s no threads left free, ASP.NET cannot execute requests.Requests are queued in a waiting queue,and only when a worker thread becomes free does a request from the queue gets the chance to execute.

When web service calls perform long I/O operations,like calls to an external web service,long running queries in the data base,or long file operations,the thread that is executing the request is occupied until the I/O operation completes.So, if such long requests are made more frequently than they complete execution,the ASP.NET worker thread pool will be exhausted.Which means further requests will be queued in the application queue and your web site will become nonresponsive for some time.

Fetching Flickr photo widget’s XML takes a couple of seconds. So, when hundreds of users load the Flickr photo widget concurrently, too many web service calls will get stuck while fetching the XML. If Flickr somehow becomes slow and takes 10 seconds to complete each request,all such proxy web service calls will get stuck for 10 seconds as well.If there’s high traffic during these 10 seconds, ASP.NET will run out of worker threads, and it will not execute new requests and the site will appear very slow to users.Moreover, if requests in the application queue are stuck for more than 30 seconds, they will time out and the site will become nonresponsive to users.

In Figure ,you can see a production server’s state when it has exceeded its limit.External web service calls are taking too long to execute, which makes the request execution time too high.Some requests are taking more than 200 seconds to complete. As a result,72 requests are stuck in calling external services,and additional incoming requests are getting queued in the application queue.

The number of requests completing successfully per second is very low as shown in the Requests/Sec counter. Also,requests are waiting in the queue for more than 82 seconds to get a free worker thread.

When there are too many requests for ASP.NET to handle, Requests In Application Queue starts to increase and Requests/Sec decreases. High Request Execution Time shows how long external web services requests are stuck.

When there are too many requests for ASP.NET to handle, Requests In Application Queue starts to increase

When there are too many requests for ASP.NET to handle, Requests In Application Queue starts to increase

Problem:A popular widget took too long to execute,the web servers got stuck, and the web site was unusable.

Solution:Changed the proxy web service to an asynchronous HTTP handler.

One time at Pageflakes, the external stock quote service was taking too long to execute.Our web servers were all getting stuck. After they were restarted, the web servers would get stuck again within 10 minutes.

The stock quote widget is very popular,and thousands of users have that widget on their page. So, as soon as they visited their page, the stock quote widget made a call via our proxy web service to fetch data from the external stock quote service. Requests to the proxy got stuck because the external web service was neither executing quickly nor timing out.Moreover, we had to use a high timeout because the external stock quote service is generally very slow.

As a result, when we had a large traffic spike during the morning in the U.S.,all our web servers repeatedly got stuck,and the web site became unusable.We had no choice but to disable the stock quote widget for an immediate solution.For a longterm solution,we had to change the stock quote proxy web service to an asynchronous HTTP handler because ASP.NET AJAX does not support asynchronous web methods.

Finding out what caused the web servers to get stuck was rather difficult. We went through the HTTP.sys error logs that are found under C: windows system32LogfilesHTTPERR. The log files were full of time out entries on many different URLs including the stock quote service URL. So, we had to turn off each URL one at a time to figure out which one was the real culprit.

All rights reserved © 2020 Wisdom IT Services India Pvt. Ltd Protection Status

ASP.NET Topics