Prevent app restarts and site downtime when deploying files

1 comments

After a fair amount of searching for the wrong keywords, I finally found two solutions.

 

Solution 1 - httpRuntime > waitChangeNotification setting

The waitChangeNotification indicates how many seconds we should wait for a new change notification before the next request triggers an appdomain restart. They both default to zero and if you set them to say 10 it will not keep restarting the app every time you FTP up a file - instead it waits 10 seconds after the last file update.

 

Snippet from http://blogs.msdn.com/b/tess/archive/2006/08/02/asp-net-case-study-lost-session-variables-and-appdomain-recycles.aspx

 

Web site updates while the web server is under moderate to heavy load

 

Picture this scenario:  You have an application with 10 assemblies in the bin directory  a.dll, b.dll, c.dll etc. (all with the version number 1.00.00). Now you need to update some of the assemblies to your new and improved version 1.00.12, and you do so while the application is still under heavy load because we have this great feature allowing you to update assemblies on the go…  well, think again...

 

Say you update 7 of the 10 assemblies and for simplicity lets say this takes about 7 seconds, and in those 7 seconds you have 3 requests come in… then you may have a situation that looks something like this…

 

Sec 1.           a.dll and b.dll are update to v 1.00.12    - appdomain unload started (any pending requests will finish before it is completely unloaded)

Sec 2.           Request1 comes in and loads a new appdomain with 2 out of 7 of the dlls updated

Sec 3.           c.dll is updated                                    - appdomain unload started         (any pending requests will finish before it is completely unloaded)

Sec 4.           d.dll is updated

Sec 5.           Request2 comes in and loads a new appdomain, now with 4 out of 7 dlls updated

Sec 6.           e.dll and f.dll is updated                       - appdomain unload started         (any pending requests will finish before it is completely unloaded)

Sec 7.           f.dll is updated

Sec 8.           Request3 comes in and loads a new appdomain with all 7 dlls updated

 

So, many bad things happened here…

 

First off you had 3 application domain restarts while you probably thought you would only have one, because asp.net has no way of knowing when you are done.  Secondly we got a situation where Request1 and Request2 were executing with partially updated dlls, which may generate a whole new set of exceptions if the dlls depend on updates in the other new dlls, I think you get the picture…  And thirdly you may get exceptions like “Cannot access file AssemblyName because it is being used by another process” because the dlls are locked during shadow copying.  http://support.microsoft.com/kb/810281

 

In other words, don’t batch update during load…

 

So, is this feature completely worthless?  No… if you want to update one dll, none of the problems above occur… and if you update under low or no load you are not likely to run into any of the above issues, so in that case you save yourself an IIS restart… but if you want to update in bulk you should first take the application offline.

 

There is a way to get around it, if you absolutely, positively need to update under load, and it is outlined in the kb article mentioned above…

 

In 1.1 we introduced two new config settings called <httpRuntime waitChangeNotification= /> and <httpRuntime maxWaitChangeNotification= />. 

 

The waitChangeNotification indicates how many seconds we should wait for a new change notification before the next request triggers an appdomain restart. I.e. if we have a dll updated at second 1, and then a new one at second 3, and our waitChangeNotification is set to 5… we would wait until second 8 (first 1+5, and then changed to 3+5) before a new request would get a new domain, so a request at second 2 would simply continue using the old domain. (The time is sliding so it is always 5 seconds from the last change)

 

The maxWaitChangeNotification indicates the maximum number of seconds to wait from the first request. If we set this to 10 in the case where we update at second 1 and 3, we would still get a new domain if a request came in at second 8 since the waitChangeNotification expired. If we set this to 6 however, we would get a new domain already if a request came in at second 7, since the maxWaitChangeNotification had then expired.  So this is an absolute expiration rather than a sliding… and we will recycle at the earliest of the maxWaitChangeNotification and waitChangeNotification.

 

There's a small issue with IIS7...

IIS7 immediately shuts down the application domain when it detects changes to the root web.config file, and ignores the httpRuntime > waitChangeNotification and maxWaitChangeNotification settings. It still respects them for changes to other files in the application, like the Bin folder ... so this is not so bad. In other words you should copy the web config last if possible. This is only a problem for app pools in Integrated Mode and not in Classic Mode.

 

Set them to never?

You could set these to a very large number to prevent all this kind of automatic restart malarchy.

<httpRuntime 
    waitChangeNotification="315360000" 
    maxWaitChangeNotification="315360000" 
/> 

However, you need some way of restarting the app after deploying or the new DLLs (or App_Code code) will never actually run. See below for a solution.

 

BTW this only affects BIN folder and App_Code folder (and a couple of other asp.net standard folders like App_Resources). It is different to aspx/aspx.cs files (and classic ASP) where every time you make a request the files are compiled if not already (which is much better .

 

Solution 2 - Turn off change monitoring 

You can turn off the change monitoring which causes the app to recompile itself. You could therefore also presumably get the app to monitor some other trigger to get it to restart when you want it to.

 

A detailed account of what is monitored is http://blogs.msdn.com/b/tmarq/archive/2007/11/02/asp-net-file-change-notifications-exactly-which-files-and-directories-are-monitored.aspx

 

 

Option A: Setting what the FileChangesMonitor monitors

This code turns off monitoring of folder deletions (why on earth do they cause restarts anyway!?). 

 

From http://www.dominicpettifer.co.uk/Blog/36/stop-iis-appdomain-restarts-when-a-folder-is-deleted?replyId=124&replyWithQuote=True

 

using System.Reflection;
using System.Web;

namespace MyWebsite
{
    /// <summary>
    ///     Stops the ASP.NET AppDomain being restarted (which clears
    ///     Session state, Cache etc.) whenever a folder is deleted.
    /// </summary>
    public class StopAppDomainRestartOnFolderDeleteModule : IHttpModule
    {
        public void Init(HttpApplication context)
        {
            PropertyInfo p = typeof(HttpRuntime).GetProperty("FileChangesMonitor",
                BindingFlags.NonPublic | BindingFlags.Public | BindingFlags.Static);

            object o = p.GetValue(null, null);

            FieldInfo f = o.GetType().GetField("_dirMonSubdirs",
                BindingFlags.Instance | BindingFlags.NonPublic | BindingFlags.IgnoreCase);

            object monitor = f.GetValue(o);

            MethodInfo m = monitor.GetType().GetMethod("StopMonitoring",
                BindingFlags.Instance | BindingFlags.NonPublic);
           
            m.Invoke(monitor, new object[] { });
        }

        public void Dispose() { }
    }
}

 

Option B: Turning off the FileChangesMonitor completely

 

From http://stackoverflow.com/questions/613824/how-to-prevent-an-asp-net-application-restarting-when-the-web-config-is-modified/629876#629876

 

internal static class HttpInternals 

    private static readonly FieldInfo s_TheRuntime = typeof(HttpRuntime).GetField("_theRuntime", BindingFlags.NonPublic | BindingFlags.Static); 
 
    private static readonly FieldInfo s_FileChangesMonitor = typeof(HttpRuntime).GetField("_fcm", BindingFlags.NonPublic | BindingFlags.Instance); 
    private static readonly MethodInfo s_FileChangesMonitorStop = s_FileChangesMonitor.FieldType.GetMethod("Stop", BindingFlags.NonPublic | BindingFlags.Instance); 
 
    private static object HttpRuntime 
    { 
        get 
        { 
            return s_TheRuntime.GetValue(null); 
        } 
    } 
 
    private static object FileChangesMonitor 
    { 
        get 
        { 
            return s_FileChangesMonitor.GetValue(HttpRuntime); 
        } 
    } 
 
    public static void StopFileMonitoring() 
    { 
        s_FileChangesMonitorStop.Invoke(FileChangesMonitor, null); 
    } 

 

Or, that code in a single method:

System.Reflection.PropertyInfo p = default(System.Reflection.PropertyInfo);
object o = null;
System.Reflection.MethodInfo m = default(System.Reflection.MethodInfo);

p = typeof(HttpRuntime).GetProperty("FileChangesMonitor", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Public | System.Reflection.BindingFlags.Static);
o = p.GetValue(null, null);
m = o.GetType().GetMethod("StopMonitoring",System.Reflection.BindingFlags.Instance |System.Reflection.BindingFlags.NonPublic);
m.Invoke(o, new object[] { });

 

Restart IIS application pool from ASP.NET page

From http://terrapinstation.wordpress.com/2008/06/12/restart-iis-application-pool-from-aspnet-page/

 

This code enables you to stop and start your application pool from the comfort of your own browser. It also gives you your application pool’s status by monitoring AppPoolState.

 

IIS Application Pool restart .aspx page

<%@ Page Language="C#" AutoEventWireup="true" CodeBehind="iis.aspx.cs" Inherits="service.iis" %>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" >
<head runat="server">
    <title>IIS App Restart</title>
</head>
<body>
    <form id="form1" runat="server">
    <div>
        Status: <asp:Label ID="lblStatus" runat="server" /><br/>
        <asp:Button ID="btnStop" Text="STOP App Pool" BackColor="IndianRed" ForeColor="White" runat="server" CommandArgument="dev.somesite.com" OnClick="stopAppPool" /><br />
        <asp:Button ID="btnStart" Text="START App Pool" BackColor="Lime" runat="server" CommandArgument="dev.somesite.com" OnClick="startAppPool" /><br />
    </div>
    </form>
</body>
</html>

Remember to replace “dev.somesite.com” in the CommandArgument attribute of the two buttons with the name of your application pool.

 

Codebehind .aspx.cs file

using System;
using System.Web;
using System.Web.UI;
using System.Management;
using System.DirectoryServices;
using System.Web.UI.WebControls;

public partial class iis : System.Web.UI.Page
{
    protected void Page_Load(object sender, EventArgs e)
    {
        Response.Write(System.Environment.MachineName);
        status();
    }

    protected void status()
    {
        string appPoolName = "dev.somesite.com";
        string appPoolPath = @"IIS://" + System.Environment.MachineName + "/W3SVC/AppPools/" + appPoolName;
        int intStatus = 0;
        try
        {
            DirectoryEntry w3svc = new DirectoryEntry(appPoolPath);
            intStatus = (int)w3svc.InvokeGet("AppPoolState");
            switch (intStatus)
            {
                case 2:
                    lblStatus.Text = "Running";
                    break;
                case 4:
                    lblStatus.Text = "Stopped";
                    break;
                default:
                    lblStatus.Text = "Unknown";
                    break;
            }
        }
        catch (Exception ex)
        {
            Response.Write(ex.ToString());
        }
    }
    protected void stopAppPool(object sender, EventArgs e)
    {
        Button btn = (Button)sender;
        string appPoolName = btn.CommandArgument;
        string appPoolPath = @"IIS://" + System.Environment.MachineName + "/W3SVC/AppPools/" + appPoolName;
        try
        {
            DirectoryEntry w3svc = new DirectoryEntry(appPoolPath);
            w3svc.Invoke("Stop", null);
            status();
        }
        catch (Exception ex)
        {
            Response.Write(ex.ToString());
        }
    }

    protected void startAppPool(object sender, EventArgs e)
    {
        Button btn = (Button)sender;
        string appPoolName = btn.CommandArgument;
        string appPoolPath = @"IIS://" + System.Environment.MachineName + "/W3SVC/AppPools/" + appPoolName;
        try
        {
            DirectoryEntry w3svc = new DirectoryEntry(appPoolPath);
            w3svc.Invoke("Start", null);
            status();
        }
        catch (Exception ex)
        {
            Response.Write(ex.ToString());
        }
    }
}

You should probably stick this little page on a separate site using a different application pool :)

 

IIS

Comments


Leave a Comment