18

To do this in-editor you open the automation tab, connect to the session and choose which tests to run.

How do you do it from the command line?

(NB. not compiling UnrealEngine/Engine/Build/BatchFiles/* comprehensively covers both building the application and compiling it. Specifically, given that you have code that is 100% happy to compile, how do you kick the test suite off)

--

Here's some more info, from recent testing on 4.10:

Running tests from the editor:

UE4Editor Project.uproject -ExecCmds="Automation RunTests MyTest"

Notice the absence of the -Game flag; this launches the Editor and runs the tests successfully in the editor console.

Running the game engine and using the 'popup log window':

UE4Editor Project.uproject -Game -ExecCmds="Automation RunTests MyTest" -log

This runs the game in 'play' mode, pops up an editor window; however, the logs stop at:

LogAssetRegistry: FAssetRegistry took 0.0004 seconds to start up

...and the game never closes or executes the tests.

Running the game engine and logging to a file:

UE4Editor Project.uproject -Game -ExecCmds="Automation RunTests MyTest" -log=Log.txt

This runs the game in 'play' mode, and then stops and never exists.

It does not appear to run any tests or log to any files.

The folder Saved/Logs does not exist after quitting the running game.

Running in the editor, test types, etc...

see: https://answers.unrealengine.com/questions/358821/hot-reload-does-not-re-compile-automation-tests.html,

Hot reload is not supported for tests; so this isn't an option.

There's also been some suggestion in various places that the test type (eg. ATF_Game, ATF_Editor) has some affect on if runs are or can be run; perhaps this is an issue to, but I've tried all kind of combinations with no success.

--

I've tried all kinds of combinations of things trying to get this working, with no success so it's time for a bounty.

I'll accept an answer which reliably:

  • Executes a specific test from the command line
  • Logs the output from that test to a file
Doug
  • 26,182
  • 26
  • 140
  • 197
  • 1
    I remember having some problems with this. I can't remeber what state I left it in but I think [this question on UE Answers](https://answers.unrealengine.com/questions/106978/run-automated-testing-from-command-line.html) contains all I ever found out. Seems like we got all tests running but not a subset of them. – T. Kiley May 31 '15 at 15:21

3 Answers3

13

Right, no one has any idea here or on the issue tracker.

After some serious digging through the UE4 source code, here's the actual deal, which I leave here for the next suffering soul who can't figure this out:

To run tests from the command line, and log the output and exit after the test run use:

UE4Editor.exe path/to/project/TestProject.uproject
              -ExecCmds="Automation RunTests SourceTests"
              -unattended
              -nopause
              -testexit="Automation Test Queue Empty"
              -log=output.txt
              -game

On OSX use UE4Editor.app/Contents/MacOS/UE4Editor.

Notice that the logs will, regardless of what you supply, ultimately be placed in:

WindowsNoEditor/TestProject/Saved/Logs/output.txt

or

~/Library/Logs/TestProject/output.txt

Notice that for mac this is outside of your project directory, in, for example, /Users/doug/Library/Logs/TestProject. (Who thought that was a good idea?)

(see https://wiki.unrealengine.com/Locating_Project_Logs#Game_Logs)

You can list automation tests using:

-ExecCmds="Automation List"

...and then parse the response to find tests to run; automation commands may be chained, for example:

-ExecCmds="Automation List, Automation RunAll"
Doug
  • 26,182
  • 26
  • 140
  • 197
  • 1
    I used this command, but when I look in the log it only says `...Automation Test Queue Empty 184 tests performed.`. I am certain that one of the tests failes and should show an error, because I added `AddError("Wrong")`. But it doesn't show up in the log. How did you find the information of tests passed and failed? – Marnix Jul 13 '17 at 08:04
  • Is there a way to run the tests with the UE Editor in "Play as Client" mode? – Jorge Luque Apr 03 '21 at 16:25
2

Do you mean the in-editor command line or the Windows command line?

In the editor you can use the Automation command with parameters, e.g. Automation RunAll

In the Windows command line you can specify unreal command parameters with -ExecCmds. To run all tests in your project: UE4Editor.exe YOURPROJECT -Game -ExecCmds="Automation RunAll"

TheBrain
  • 597
  • 7
  • 11
  • Doesn't seem to do anything? It just opens the project in game mode. Should it print out the test results to the command line or something? – Doug Mar 19 '15 at 09:06
  • It should print it to a log file in YourProjectDir/Saved/Logs. – TheBrain Mar 19 '15 at 09:40
  • Only files in my Saved folder are AutoScreenshot.png, Autosaves, Backup, Config :( – Doug Mar 19 '15 at 12:21
0

For anyone still wondering, there is a bug in the editor that make it so the test list is flushed before they are run when they are started from the command-line (be it at startup or after).

This means that the editor actually compiles a list of tests to run, which is then flushed by another part of the program. The editor then thinks that it has finished running all the test and, since there is no errors, shows that they all succeeded.

I can post how to do a fix to this if anyone is interested, but it introduce another minor bug.

Karmoka
  • 11
  • Karmoka, this answer does not address any of the concerns raised in the question. While this is helpful, it is best framed as an addon to an answer to the question. – Jared Clemence Dec 21 '17 at 19:51
  • 1
    It does answer the question, since it is quite literal impossible to run the tests from the command line as of UE4.16 without changes to the engine. By its nature, no arguments can bypass this bug. – Karmoka Dec 22 '17 at 14:56
  • 1
    Starting your answer with something like "It is quite literally impossible to run the tests from the command line" would help make your answer more easily readable. That is a good edit that you have suggested. – Jared Clemence Dec 22 '17 at 16:46