2

I am using SpringBoot with Cucumber. One particular test case fails when run together and passes when run individually. Other Scenarios in the same feature file seems to work fine. I tried the following links but they did not work.

  • link1, link2, link3, link4, link5
  • I also tried to quarantine the problematic Scenario to a separate feature file and wrote an exclusive step def for the feature, but still no luck.
  • I tried introducing @After and @Before hooks to reset the value of Instance variable in stepdef file to null.
  • I even tried switching the position of Sample-payload and id Count Set

Now I ran out of options. Please help.

Feature File:

Scenario Outline: To check if the service returns all the ids when more than 10000 Ids are returned by the query
Given sample request <payload>
 When the new end point is invoked
 Then the service returns <idCount> Ids

Examples: 
  | payload          | idCount | 
  | sample-payload-1 | 17575   | 
  | sample-payload-2 | 4       | 
  | sample-payload-3 | 23535   | 
  | sample-payload-4 | 34535   |

Step Def File:

public class MyStepdefs extends AbstractStepsDefs {

@Autowired
MyController myController;

private String requestPayload;

private List<String> ids;

@Given("^sample request (.+)$")
public void sample_request_something(String payload) throws Throwable {
    this.requestPayload = payload;
}

@When("^the new end point is invoked$")
public void the_new_end_point_is_invoked() throws Throwable {
    String responseJSON = MyUtil.getPostResponseJSON(myController, "/ids", requestPayload);
    responseJSON = responseJSON.replace("[", "").replace("]", "").replaceAll("\"", "");
    ids = Arrays.asList(responseJSON.split(","));
}

@Then("^service returns list of available (.+)$")
public void service_returns_list_of_available_something(String ids) throws Throwable {
    List<String> list = Arrays.asList(ids.split(","));
    Assert.assertTrue(this.ids.containsAll(list));
}

@Then("^the service returns (.+) ids$")
public void the_service_returns_ids(String idCount) throws Throwable {
    System.out.println("Actual Size:" + this.ids.size());
    System.out.println("Expected Size:" + Integer.parseInt(idCount));
    Assert.assertEquals(this.ids.size(), Integer.parseInt(idCount));
}

@After
@Before
public void cleanUp() {
    ids = null;
}

}

Now I have run out of options. Please help.

UPDATE1: The second then block in the stepdef class fails. The sample-payload-1 and sample-payload-2 pass, but the other two fails. Even after changing the order of the sample-payloads the tests fail.

The error I get is an Assertion error as the size of the ids list is not same. But when I run the same test Individually I don't get this error as the size matches.

Andy
  • 2,911
  • 5
  • 17
  • 30
  • 1
    Which one of the test case fails in the ScenarioOutline? Can you clarify? What is the error? – Grasshopper Jun 14 '18 at 19:02
  • @Grasshopper: I have addressed your question in the update1 section. Thanks. – Andy Jun 14 '18 at 19:15
  • Inside the method the_new_end_point_is_invoked that corresponds to When, print out the size of the `ids` immediately after it is returned. If there is a discrepancy then it is not your problem, which anyways seems to be the case. Guessing data returned has some other logic for multiple repeats or simply the count has changed. – Grasshopper Jun 14 '18 at 19:35
  • The size in the when block is not correct. I just checked. :-( – Andy Jun 14 '18 at 19:48
  • Then it is not ur problem. Investigate ur testing data. Maybe things have changed in the backend. – Grasshopper Jun 14 '18 at 19:50
  • But if testing data is wrong it should not pass when run individually, right? Also, the same print statement returns the correct size when run individually. – Andy Jun 14 '18 at 19:53
  • Guessing, but is there some kind of dependency between calls to this api? Or maybe caching? Only easy way is to get hold of the developer. What is the difference in numbers? Are these ids been returned in the earlier calls also? – Grasshopper Jun 14 '18 at 19:59
  • I am the one that developed the API. All are independent rest calls. There is no caching. The service fires a query or queries and fetches all the records. So if the actual no of records is 34535, the service runs the query 4 times to get the ids in the following fashion: 10k, 10k, 10k, 3453. When I run individually the number of records I get is 34535. When I run together I get only 10K. FYI, I have used recursion in the service implementation. Wondering if this will cause any error. – Andy Jun 14 '18 at 20:08
  • 2
    Seems to be one call logic is affecting other call. Maybe the recursion check condition needs to be looked at. Not experienced in rest at all, guessing here but maybe some code is not thread safe. – Grasshopper Jun 14 '18 at 20:15
  • Thanks a lot for the hint. and your time. It works fine now. Please find my answer. – Andy Jun 14 '18 at 21:21

1 Answers1

1

The issue was with the Instance variable I had used as a counter to loop through a logic that fires the query and extracts the result. The first test sample [sample-payload-1] will alone work fine as it has a fresh copy of the instance counter. The subsequent samples will have only the altered counter value.

This explains why the test cases passed when ran individually as there is not another test that would have used the altered counter.

FIX: The fix was to reset the counter back to 0 before I exit the implementation/service class so that the subsequent request will have a fresh counter

Lesson Learnt: Never rely on an instance variable if the scope of the enclosing Class is Singleton. The value will keep changing for every call to the method in the Class

Andy
  • 2,911
  • 5
  • 17
  • 30
  • 1
    Happy you found the problem! Fyi: you can use cucumber-spring (or several other DI frameworks, but you already have spring) to manage state and reset it between scenarios. I did a brief write up of it [here](https://medium.com/@mlvandijk/managing-state-in-cucumber-jvm-using-spring-a795e9a1dd18). We still need to update the [docs](https://docs.cucumber.io/) with some of this info. – Marit Jun 17 '18 at 05:30