Browserstack reports successful even when test fails in Nightwatchjs

Question

I just started using nightwatch with browserstack and I'm noticing that when we get a failed test, nightwatch registers the failure, but browserstack does not. sample test I am using. Also I am using free trial version of BrowserStack.

My question is:

Are there any ideas how to tell browserstack when a test run failed ?

From BrowserStack doc:

REST API

It is possible to mark tests as either a pass or a fail, using the following snippet:

var request = require("request"); 
request({
    uri: "https://user:key@www.browserstack.com/automate/sessions/<session-id>.json",
    method: "PUT",
    form: {
        "status": "completed",
        "reason":""
    }
});

The two potential values for status can either be completed or error. Optionally, a reason can also be passed.

My questions are:

How I can get 'session-id' after test execution ?
What if I can see "completed" status in dashboard already ?

score 6 · Accepted Answer · edited Jul 16 '17 at 16:08

6

A session on BrowserStack has only three types of statuses: Completed, Error or Timeout. Selenium (and hence, BrowserStack) does not have a way of understanding, if a test has passed or failed. Its by the multiple assertions in your tests that appear on your console, that you infer if a test has passed / failed. These assertions however, do not reach BrowserStack. As you rightly identified, you can use the REST-API, to change the status of the session to 'Error', if you see a failure in your console.
I would suggest fetching the session ID of the test as the test is being executed, since fetching the session ID after the test execution is a lengthy process. In Nightwatch, you can fetch session ID as follows:

browser.session(function(session) {
    console.log(session.sessionId);
});

Yes, you can certainly change the status of the session once it is completed. That's where the REST-API comes to help!

edited Jul 16 '17 at 16:08

Luciano van der Veekens

5,586
4
21
29

answered Jan 30 '16 at 13:00

Umang Sardesai

742
5
13

2

Is there an easier way to force fail the test at BrowserStack? Instead of having to meddle with the API, figuring out the build and session ID... – dialex Jun 06 '16 at 16:34
2

You can mark the session as "Error" only via REST-API. There is no other approach. That said, you don't need the build ID to change the status of a session, only the session ID is enough. Check this link for more details - https://www.browserstack.com/automate/rest-api#rest-api-sessions – Umang Sardesai Jun 06 '16 at 19:02
1

Their support told me that I could do `driver.getSessionId()` to get the id right away. – dialex Jun 07 '16 at 08:33
Depends on what language you are using. If you are using Nightwatch, you can use the method shared above. For Java, it is `driver.getSessionId()` to get the session ID. – Umang Sardesai Jun 07 '16 at 10:21

score 1 · Answer 2 · answered Jun 20 '19 at 12:39

If you came here searching for a solution in Python, you could use

requests.put(
    "https://api.browserstack.com/automate/sessions/{}.json".format(driver.session_id),
    auth=(USERNAME, ACCESS_KEY),
    json={"status": "failed", "reason": "test failed"})

Browserstack reports successful even when test fails in Nightwatchjs

2 Answers2

Linked