Testng 6.9.10 assertEquals() not failing like it should

Question

So I set up a Test case using TestNG. When I do an assert equals on something that should fail, the test passes anyway. I have tried comparing them as list and string. The actual comes out as:

actual = [The license key was not valid., 
          The license key has expired and is no longer valid., 
          That email domain is not allowed., 
          The maximum number of allowed users for that license key has been reached. No other users may register unless a user is removed or the user limit for that key is increased., 
          That license key does not have a valid start date, the administrator must define an effective date for the license key.]

The code that I am using is below. Actual and expected are clearly different but the test still passes. I am using TestNG 6.9.10. I haven't been able to find anywhere that there is a bug with assertEquals.

import org.testng.annotations.*;
import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertTrue;

@Test
public void verifyLicencingErrorsTest() {
    List<String> expected = Arrays.asList(
            "The license key was not valid.",
            "The license key has expired and is no longer valid.",
            "The license key was not valid.",
            "The maximum number of allowed users for that license key has been reached. No other users may register unless a user is removed or the user limit for that key is increased.",
            "That license key does not have a valid start date, the administrator must define an effective date for the license key."
    );
    List<String> actual = createAccountPage.verifyLicencingErrors();

    try {
        assertEquals(actual.toString(), expected.toString(), "The actual alerts did not match what was expected");
    } catch (Error e) {
        verificationErrors.append(e.toString());
    }
}

I think I have provide all of the information necessary information. If I need more please let me know.

Edit: Added imports

What are your `import`s? Also: you are putting your assert inside of a try..catch? Why? That will never fail! — SiKing, Mar 29 '16 at 22:51
@SiKing went ahead and added the imports to the code block. I see now. I originally had this set up with JUnit and TestNG. JUnit would throw exceptions and stop my tests all together so I had to add the try/catch. Looks like I will have to change it all to make this work correctly. Thank you for the reply! — Michael Gibson, Mar 29 '16 at 23:02
It's unlikely that an exception thrown by the framework, be it JUnit or TestNG, will stop an entire suite. Normally it should run all of them and in the end report the number of test run, failed and skipped. Either way you [should not be catching `Error`](http://stackoverflow.com/questions/352780/when-to-catch-java-lang-error), at the very least catch `Exception` if you really need to, but even these should be special-corner cases or the purpose of the method should be to test that the exception is thrown, case in which you can use the `expectedExceptions` feature of the `@Test` annotation. — Morfic, Mar 30 '16 at 10:15
@SiKing regarding the `try-catch` block, I think the correct statement may be something along the lines of: _That is supposed to fail when the values don't match by the means of an AssertionError_ :-) — Morfic, Mar 30 '16 at 10:19

Morfic · Answer 1 · 2016-03-30T11:38:08.310

As per the TestNG Assert documentation:

public static void assertEquals(java.lang.String actual,
                                java.lang.String expected,
                                java.lang.String message)

Asserts that two Strings are equal. If they are not, an AssertionError, with the given message, is thrown.
Parameters:
actual - the actual value
expected - the expected value
message - the assertion error message

The issue is that you're catching Error, thus the thrown AssertionError no longer reaches the framework and the test is considered ok. If you log the error you catch you can easily see that:

    try {
        assertEquals(actual.toString(), expected.toString(), "The actual alerts did not match what was expected");
    } catch (Error e) {
        log.error("Caught an [" + e.getClass().getName() + "]", e);
    }

... yelds:

[ERROR] 2016-03-30 12:51:01,345 test.LauncherTest - Caught an [java.lang.AssertionError]
java.lang.AssertionError: The actual alerts did not match what was expected expected [[The license key was not valid., The license key has expired and is no longer valid., The license key was not valid., The maximum number of allowed users for that license key has been reached. No other users may register unless a user is removed or the user limit for that key is increased., That license key does not have a valid start date, the administrator must define an effective date for the license key.]] but found [[The license key was not valid., The license key has expired and is no longer valid., That email domain is not allowed., The maximum number of allowed users for that license key has been reached. No other users may register unless a user is removed or the user limit for that key is increased., That license key does not have a valid start date, the administrator must define an effective date for the license key]]
    at org.testng.Assert.fail(Assert.java:94)
    at org.testng.Assert.failNotEquals(Assert.java:496)
    at org.testng.Assert.assertEquals(Assert.java:125)
    at org.testng.Assert.assertEquals(Assert.java:178)
    at test.LauncherTest.verifyLicencingErrorsTest(LauncherTest.java:42)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:85)
    at org.testng.internal.Invoker.invokeMethod(Invoker.java:639)
    at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:821)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1131)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:124)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:108)
    at org.testng.TestRunner.privateRun(TestRunner.java:773)
    at org.testng.TestRunner.run(TestRunner.java:623)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:357)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:352)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:310)
    at org.testng.SuiteRunner.run(SuiteRunner.java:259)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1185)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1110)
    at org.testng.TestNG.run(TestNG.java:1018)
    at org.testng.remote.RemoteTestNG.run(RemoteTestNG.java:111)
    at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:204)
    at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:175)
    at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:125)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)

While removing the catch-block, or potentially refining the exceptions caught, will work as expected:

java.lang.AssertionError: The actual alerts did not match what was expected 
Expected :[The license key was not valid., The license key has expired and is no longer valid., The license key was not valid., The maximum number of allowed users for that license key has been reached. No other users may register unless a user is removed or the user limit for that key is increased., That license key does not have a valid start date, the administrator must define an effective date for the license key.]
Actual   :[The license key was not valid., The license key has expired and is no longer valid., That email domain is not allowed., The maximum number of allowed users for that license key has been reached. No other users may register unless a user is removed or the user limit for that key is increased., That license key does not have a valid start date, the administrator must define an effective date for the license key]
  <Click to see difference>


    at org.testng.Assert.fail(Assert.java:94)
    at org.testng.Assert.failNotEquals(Assert.java:496)
    at org.testng.Assert.assertEquals(Assert.java:125)
    at org.testng.Assert.assertEquals(Assert.java:178)
    at test.LauncherTest.verifyLicencingErrorsTest(LauncherTest.java:41)

===============================================
Custom suite
Total tests run: 1, Failures: 1, Skips: 0
===============================================

Testng 6.9.10 assertEquals() not failing like it should

1 Answers1