5

Allure report tool was designed to show one report per test run. Developers warned me about it. Anyway i need more from this convenient tool.

  • Did you think to keep allure reports for several days/month?
  • Did you think to analyze history data automatically?

Basically, I need to find answer on question: What tests are flaky for previous month and need to be stabilized? Unfortunately, I didn't find answer from the box on the main web site(http://allure.qatools.ru/) or in the internet.

My solutions:

Trying to solve first question: [Solved]

I tried to keep allure reports archived under jenkins CI tool, but i have to manage 1500 TestNg tests and allure report generation and archiving takes about 10min, which is unacceptable for our pipeline. Then i started to generate report on jenkins slave and ftp it to remote box according next structure: project_type/pipeline_type/suite_type/day/build_number.

I store reports as raw projects with btrfs compression, which allows to compress json files very effectively(files are compressed in 5 times) and files are accessible from ui.

UI is a django(simply love it) project, which follows files system structure.

This approach allows to aggregate data.

Trying to solve second question: [Not solved]

I need to aggregate data across all runs for suite_type(see structure above) and find out which tests are failed on main run or re-run attempts(we re-run our failed tests 3 times using FailedReporter TestNg listener) and match failed reasons with failed tests. Ideally i need to match failed tests results with stored reports.

For example:

We have 2 reports:

../parallel-suite/2015-10-21/560

../parallel-suite/2015-10-22/584

Each suite has failed test testCreditCard.

  • Test failed in 560 build with reason - third party system was down.
  • Test failed in 584 build on first re-run attempt with reason - third party system was down, on second re-run attempt with reason - IndexOutOfBoundsException: Index: 0, Size: 0 at java.util.ArrayList.rangeCheck, on thrid attempt it passed.

As a result i need to know that third party system was down error was 2 times for 2 days and IndexOutOfBoundsException was 1 time for 2 days and test passed on second day on third re-run attempt.

Does someone work in this direction?

Community
  • 1
  • 1
Pavel Susloparov
  • 71
  • 1
  • 4

1 Answers1

1

Yes, Allure supports aggregated reports. But it is more about:

  • distributed execution (according to Allure model every test case can be labelled with host it was executed on)
  • multiple executions (e.g. tests are written in different programming languages - you can't execute single Allure adapter).

As far as I know Allure will never support tracking historic test results, discovering trends etc. Historic test results might become enormous in amount and just crash your browser. Also timeline tab would make no sense any more. Other tabs would require significant changes. Why don't you use Sonar for those scenario?

volkovs
  • 1,003
  • 2
  • 10
  • 21
  • Thank you for the answer! It's clear now that i have to create something to aggregate tests results. We use Sonar for tracking unit tests coverage, i will take a look how to use it for integration and functional tests. – Pavel Susloparov Nov 24 '15 at 15:18
  • @volkos I am also using Allure. For me, I ran a Jenkins' job several times in the past 8 hours. So I have more than 50 builds data. Each build generates a report I want to combine results of last 50 reports to one. How can I do that? – paul May 19 '19 at 09:25
  • @paul probably you have to update both Allure library and Jenkins plugin. If run report generation via Jenkins plugin, the test case page contains `History` tab (so you could compare results of the same test at previous job executions) and `Retries` tab (for flaky tests executed multiple times within single job execution). – volkovs May 22 '19 at 06:54