DevOps with VSTS - The Second Way - Monitoring

Posted on Monday, February 6, 2017

This is part 7 of the DevOps with VSTS series

A big part of The Principles of Feedback is making sure that the system is providing enough telemetry data about the performance and usage of the system.
By having this data, analysis can be done on the whole system, but especially on newly released features.
In a DevOps world where deployments are done very quickly, it is important to know what impact a new feature has.
To know, for this feature, what the best way forward is, questions like these need to be answered:

  • Is the feature generating any errors?
  • Is the new feature being used by customers,?
  • What type of customers are using the feature?

The answers to these questions are important to know whether more development work needs to be put into the feature, the feature is completed or whether the feature should be removed entirely.

Application Insights

For me, the Go-To solution to track and analyze application telemetry is Azure Application Insights.
There are already a lot of great features in AppInsights and more are being added all the time.

Adding AppInsights to an existing application is pretty easy to do from within Visual Studio.
First right click the project and select Configure Application Insights.

This will start a nice wizard which will help in setting up AppInsights into the application.

When the wizard is completed, AppInsights is completely configured for this application.
The AppInsights client in the application is using a InstrumentationKey to send data to AppInsights.
This InstrumentationKey is configured in the ApplicationInsights.config file in the project.
To be able to easily deploy the application into multiple environments, this InstrumentationKey should be decoupled from the source code.
This can be done easily be adding a AppSetting for the InstrumentationKey and setting that during startup of the application.

If the application is a ASP.NET MVC WebApp, the InstrumentationKey is probably also in the _Layout.cshtml, change this to use the AppSetting as well.

Before deploying this updated application, remember to update the Release Definition as show in Continuous Integration to include the newly created AppSetting.

With AppInsights set up, browse to the Azure Portal and open the AppInsights instance.
Opening up the LiveStream and in another window execute some functionilty in the WebApp, LiveStream should now show telemetry about the application.

Automated Monitoring

In AppInsights it is possible to set up WebTests.
These tests will test if the WebApp is reachable and if it's responding fast enough.
If any of the test will faill, AppInsights will generate an alert which is send by email to the Azure subscription administrators.
It is also possible to add a webhook to the alert. AppInsights will then also invoke that webhook when an alert is activated or resolved.
This isn't specific to the alerts from WebTests but is possible for all alerts configured in AppInsights.

The first thing to do is to create the WebTest.
In the Azure Portal browse to the AppInsights instance, select Web tests and then Add web test.

By invoking a webhook during an alert, it is possible to automatically create a Live Site Incident in VSTS.
By using a small Azure Function and automating this, it is ensured that issues are registered in VSTS and all incidents will be tracked and analyzed, not just when someone manually creates on LSI.

First create a new Azure Function App and select the GenericWebHook-CSharp as the template.

When the Function App has been created, copy the Function Url.

Paste this url in the WebTest in AppInsights.

In the Function App, create an AppSetting named personalAccessToken. This token is used to authenticate against the VSTS API. How to get a PersonalAccessToken is shown here
There are two more AppSettings that are needed to be able to add LSI's to the backlog.

  • vstsDomain
    • This is the VSTS domain, eg.
  • vstsLsiProject
    • This is the name of the VSTS project where the LSI's should be added to.

For the implementation of the Function App first create a project.json file and copy this content:

    "frameworks": {
        "net46": {
            "dependencies": {
                "Microsoft.AspNet.WebApi.Client": "5.2.3",
                "Microsoft.AspNet.WebApi.Core": "5.2.3",
                "Microsoft.Azure.KeyVault.Core": "2.0.4",
                "Microsoft.Azure.WebJobs": "1.1.2",
                "Microsoft.Azure.WebJobs.Core": "1.1.2",
                "Microsoft.Data.Edm": "5.8.2",
                "Microsoft.Data.OData": "5.8.2",
                "Microsoft.Data.Services.Client": "5.8.2",
                "Microsoft.TeamFoundationServer.Client": "14.102.0",
                "Microsoft.VisualStudio.Services.Client": "14.102.0",
                "Microsoft.WindowsAzure.ConfigurationManager": "3.2.3",
                "Newtonsoft.Json": "9.0.1",
                "System.ComponentModel.EventBasedAsync": "4.3.0",
                "System.Dynamic.Runtime": "4.3.0",
                "System.Linq.Queryable": "4.3.0",
                "System.Net.Requests": "4.3.0",
                "System.Spatial": "5.8.2",
                "WindowsAzure.Storage": "8.0.1"

After that create a app.config file and copy this content:

<?xml version="1.0" encoding="utf-8"?>
        <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.6.2" />
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
        <assemblyIdentity name="Newtonsoft.Json" publicKeyToken="30ad4fe6b2a6aeed" culture="neutral" />
        <bindingRedirect oldVersion="" newVersion="" />
        <assemblyIdentity name="Microsoft.VisualStudio.Services.WebApi" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="" newVersion="14.102.25423.0" />
</configuration><?xml version="1.0" encoding="utf-8"?>

Next update the run.csx file and copy this:

public static async Task<object> Run(HttpRequestMessage req, TraceWriter log)
    log.Info("Start run");
    var personalAccessToken = ConfigurationManager.AppSettings["personalAccessToken"];
    var jsonContent = await req.Content.ReadAsStringAsync();
    log.Info($"Content: " + jsonContent);
    dynamic data = JsonConvert.DeserializeObject(jsonContent);

    // Azure Alert
    if (data.status != null && data.context != null)
        var alertStatus = data.status.ToString();
        var alertName =;
        var alertTimeStamp = data.context.timestamp.ToString();
        await ProcessAzureAlert(personalAccessToken, alertStatus, alertName, alertTimeStamp);
        return req.CreateResponse(HttpStatusCode.OK);

    return req.CreateResponse(HttpStatusCode.BadRequest);

Next add the method ProcessAzureAlert

private static async Task ProcessAzureAlert(string personalAccessToken, string alertStatus, string alertName, string alertTimeStamp)
    if (alertStatus == "Activated")
        var uri = new Uri($"https://{ConfigurationManager.AppSettings["vstsDomain"]}");
        var projectName = ConfigurationManager.AppSettings["vstsLsiProject"];
        var credentials = new VssBasicCredential("", personalAccessToken);
        using (var workItemTrackingHttpClient = new WorkItemTrackingHttpClient(uri, credentials))
            var lsi = new JsonPatchDocument();
            lsi.AddRange(new List<JsonPatchOperation>
                new JsonPatchOperation
                    Operation = Operation.Add,
                    Path = "/fields/System.Title",
                    Value = $"The alert {alertName} is {alertStatus}"
                new JsonPatchOperation
                    Operation = Operation.Add,
                    Path = "/fields/DevOps.IncidentSeverity",
                    Value = "1 - High"
                new JsonPatchOperation
                    Operation = Operation.Add,
                    Path = "/fields/DevOps.ServiceCategory",
                    Value = "FrontEnd"
                new JsonPatchOperation
                    Operation = Operation.Add,
                    Path = "/fields/DevOps.IncidentDetected",
                    Value = alertTimeStamp
                new JsonPatchOperation
                    Operation = Operation.Add,
                    Path = "/fields/DevOps.IncidentStarted",
                    Value = alertTimeStamp
                new JsonPatchOperation
                    Operation = Operation.Add,
                    Path = "/fields/DevOps.DetectionMethod",
                    Value = "Automated"

            await workItemTrackingHttpClient.CreateWorkItemAsync(lsi, projectName, "Live Site Incident");
    await Task.FromResult(0);

The code should be pretty self-explanatory. Every time the WebTests fail, AppInsights will trigger the WebHook, which will create a LSI in VSTS.
With the monitoring also completely set up, the next thing is to have a look at the next post The Third Way - The Principles of Continual Learning and Experimentation

Disclaimer: Any views or opinions expressed on this blog are my own personal ones and do not represent my employer in any way.