Viewing Unit Test and Coverage Reports Generated in AWS CodeBuild

Question

I am using AWS CodeBuild to run unit tests for my python project using pytest. I am using the --junitxml argument and the pytest-cov package to generate test reports and coverage reports that I've listed as artifacts in my buildspec.yml.

I've used Jenkins in the past to do this and Jenkins provides some nice graphs and tables showing test result history and coverage history as well as results from the most recent test.

Is there a good way to view the reports generated by my CodeBuild project? I haven't found anything in CodeBuild or CodePipeline directly. Do I have to use a separate tool that can ingest the report files? If so, what are some tools for this?

score 3 · Answer 1 · answered Dec 15 '19 at 22:16

CodeBuild recently announced support for test reports. Copied from the blog post:

The reports can be in the JUnit XML or Cucumber JSON format. You can view metrics such as Pass Rate %, Test Run Duration, and number of Passed versus Failed/Error test cases in one location. Builders can use any testing frameworks as long as the reports are generated in the supported formats.

Two things needs to be updated to accomplish this. First, add some configuration to the buildspec.yml file:

reports:
  SurefireReports: # CodeBuild will create a report group called "SurefireReports".
    files: #Store all of the files
      - '**/*'
    base-directory: 'target/surefire-reports' # Location of the reports

Secondly, CodeBuild needs some additional IAM permissions:

{
    "Statement": [
        {
            "Resource": "arn:aws:codebuild:your-region:your-aws-account-id:report-group/my-project-*", 
            "Effect": "Allow",
            "Action": [
                "codebuild:CreateReportGroup",
                "codebuild:CreateReport",
                "codebuild:UpdateReport",
                "codebuild:BatchPutTestCases"
            ]
        }
    ]
}

score 2 · Answer 2 · answered Oct 30 '18 at 21:49

One solution could be to let pytest generate html reports, e.g. by using pytest-html plugin (pytest-cov has native support for html report generation). Then, you can use the aws s3 cp command in your buildspec.yml file to copy the reports to an S3 bucket. Alternatively, you can attach the reports to the build artifact in the buildspec.yml file. Note, not everyone are comfortable with the idea of mixing the test reports together with the executable code. It depends on how the generated artifacts are being treated, the amount of desired post processing, how they are being delivered and deployed, if they will be consumed by a third party, etc.

I thought of this as well. The downside is that it doesn’t give nice history and visualization like on Jenkins. Do you know if any tools that could consume these reports? — Brian, Nov 01 '18 at 00:06

Viewing Unit Test and Coverage Reports Generated in AWS CodeBuild

2 Answers2