How to create parametrized performance tests in MSTest using xml?

Question

I need to generate some performance testing using MSTest (LoadTests) and Visual Studio 2010. I need to make a test of a restful service and gather metrics.

I have created an integration test that request one URL (something restful like http://hostname/get/201212 or whatever is the ID). Later I create a loadtest and manage to execute the integration test thousands of times and investigate response time and server metrics (CPU Load, Memory, etc.)

Now I need to perform a similar scenario but each test need to have a different ID read from a given XML that contains thousands. The idea is not having anything cached.

What I have tried/thought so far?

1. The "bubble-sort" solution is create thousands of tests, each one with one id. Not feasible
2. The next approach is create one test that read xml, iterate through it and make request. The problem of this approach is that I will end having a big test that takes minutes to run.
3. I have a possible workaround using [ClassInitialize] to load all IDs from XML and [TestInitialiaze] to change a global variable each test execution...

I am not sure if the last option is the best one. Is there any mechanism in MSTests to allow me to perform LoadTests using different IDs read from a XML?

Answer 1

I don't like answering my own questions, but I put this here for future reference of people with my same problem.

Why other 3rd option was wrong

The problem of 3rd option is that in a LoadTest with 1000 iterations with a list of IDs of 1000 the LoadTest was running 1000 times with the first ID instead of 1000 times, each one with each ID. I used some sort of nosql profiler to prove it.

With following solution if my Loadtest has 1000 iterations and my csv list has 1000 IDs, 1000 tests will be executed each one with one ID. If it happens that your LoadTest have more iterations, say 2000, then in LoadTest 1001 will start again from the beginning of csv list.

My Solution

NOTE: This solution uses a csv file. It can be easily adapted to use another different DataSource as xml, excel, a table in SQL server, etc.

The correct option to achieve this is creating a test with DataSource attribute. For example you create a csv file that contains the IDs you want to use in your test. Example of my csv file:

ID
1003002-20121211120000
1004071-20121211120000

You need to add a DataContext in your tests:

private TestContext testContextInstance;
public TestContext TestContext
{
    get { return testContextInstance; }
    set { testContextInstance = value; }
}

Finally, your test should be like this:

    [TestMethod,DeploymentItem("DataOrigin\\list.csv"), DataSource("Microsoft.VisualStudio.TestTools.DataSource.CSV", "|DataDirectory|\\list.csv", "list#csv", DataAccessMethod.Sequential)]
    public void TestScenario_1_DATADRIVEN()
    {
        // PREPARATION
        string ID = TestContext.DataRow["ID"].ToString();
        string querystring = CreateQueryWithErrorDebug(ID);

        // EXECUTION
        string result = RunXCCQuery(querystring);

        // ASSERTS
        Assert.IsTrue(result.Length > 0);
        Assert.IsTrue(result.Contains(ID));
    }

DeploymentItem attribute copies csv file. DataSource attribute read it and iterate thru it.

This line is the place where your test reads from TestContext.

string ID = TestContext.DataRow["ID"].ToString();

Having n IDs in the CSV file will cause the test to be run n times, one per ID.

Some useful links I have found in my investigation:

• MSDN Info
• This question in StackOverflow
• MSDN explanation on how to access different DataSources

Answer 2

The "bubble-sort" solution is create thousands of tests, each one with one id. Not feasible

Totally agreed. It cannot be done.

The next approach is create one test that read xml, iterate through it and make request. The problem of this approach is that I will end having a big test that takes minutes to run.

This approach is not load testing. You will have only one test sending one request at a time.

I think the best a approach is the 3rd one with some variations. The way I solved a similar problem was by:

1. Creating a class (let's say DataInput) that will provide me different data for each test.
2. The DataInput is reading data from scv files and has static methods providing the data I want (e.g. getNextUserName).
3. In the [ClassIntialize] I create a static DataInput object.
4. In the [TestInitialize] I am creating the actual request to the rest service by getting data from the DataInput.

That way all the tests have different data.