Automating the license checking in gitlab CI/CD pipeline

Question

I am trying to implement the license checking of the installed packages in my react project on a GitLab CI/CD pipeline. Before the build stage, the pipeline should check all the available licenses and then whitelist or blacklist certain specified licenses.

I am using the license-checker package to implement a list of available licenses in a JSON file. After executing the required command: license-checker --json > ./license.json, the output is:

license.json

{
  "@babel/plugin-transform-parameters@7.10.1": {
    "licenses": "MIT",
    "repository": "https://github.com/babel/babel",
    "path": "..../node_modules/@babel/plugin-transform-parameters",
    "licenseFile": "...../node_modules/@babel/plugin-transform-parameters/LICENSE"
  },
  "@babel/plugin-transform-property-literals@7.10.1": {
    "licenses": "MIT",
    "repository": "https://github.com/babel/babel",
    "path": "..../node_modules/@babel/plugin-transform-property-literals",
    "licenseFile": "...../node_modules/@babel/plugin-transform-property-literals/LICENSE"
  },
  "@babel/plugin-transform-react-constant-elements@7.10.1": {
    "licenses": "MIT",
    "repository": "https://github.com/babel/babel",
    "path": "..../node_modules/@babel/plugin-transform-react-constant-elements",
    "licenseFile": "...../node_modules/@babel/plugin-transform-react-constant-elements/LICENSE"
  }
  // .........and list goes on
}

.gitlab-ci.yml

include:
  - local: license-checker-config.yml

stages: 
  - dependency

dependency:
  image: node:12
  stage: dependency
  script: 
    - npm ci
    - echo "main file...."

license-checker-config.yml

before_script: 
  - ./license.json
  - echo "Checking licenses..."

License scanning should be initiated before the build process, so I included as a part of before_script. In license-checker-config.yml, I need to include my JSON file and then check by iterating over it, if it contains license like MIT, then only the build stage should continue otherwise the build should fail.

With my current code setup, I executed the pipeline and got the error:

Executing "step_script" stage of the job script
00:01
$ ./license.json
/bin/bash: line 99: ./license.json: Permission denied
ERROR: Job failed: exit code 1

Even though the file license.json exists in the same root folder, it shows permission denied. Further, I am unable to figure out how to implement JSON file looping inside the yml file and then achieve the required.

Any help to get me through this is highly appreciated.

Answer 1

Easy mode (nodejs)

You can add a script in your package.json (source code of your app) with name validate_licenses.js

  "scripts": {
    "start": "...",
    "build": "...",
    "validate_licenses": " node validate_licenses.js"
  }

Put the logic of license validation in validate_licenses.js:

Finally just execute in any part of your pipeline:

npm run validate_licenses

This will fail like a test and the build process will be interrupted.

- force json creation and parse it

const { exec } = require("child_process");
const license_checker = require('license-checker')
var fs = require('fs');

exec("license-checker --json > ./license.json", (error, stdout, stderr) => {
    if (error) {
        console.log(`error: ${error.message}`);  return;
    }
    if (stderr) {
        console.log(`stderr: ${stderr}`);return;
    }
    console.log(`json created`);
    parseJsonLicenses();
});

function parseJsonLicenses(){
  var licenses = JSON.parse(fs.readFileSync('/license.json', 'utf8'));
  //iterate licenses and fail if exist one licence differet of MIT
  for(var npmModule in licenses){
   if(licenses[npmModule].licenses != 'MIT')
     throw new Error(npmModule+' has a not allowed license');
  }  
}

More information in its official site:

• https://www.npmjs.com/package/license-checker
• https://github.com/davglass/license-checker/tree/master/tests

- avoid json generation and use [onlyAllow] option

var checker = require('license-checker');
var config = {
  start: '.' ,
  onlyAllow: 'MIT'
};
checker.init(config, function(json, err) {
    if (err) {
        throw new Error(err);
    } else {
        console.log (JSON.stringify (json))
    }
});

I tested in one of my on projects and I get this error:

Package "@csstools/convert-colors@1.4.0" is licensed under "CC0-1.0" which is not permitted by the --onlyAllow flag. Exiting.

---

Hard mode (shell commands)

- parse previously generated json

You can read, iterate an find MIT word with pure shell commands but it will be a hard task.

Check this: Iterating through JSON array in Shell script

If you succeed, you can invoke the validate_licenses.sh in your git lab ci:

test:
    stage: test
    script:
        - echo 'starting licenses validation'
        - ./validate_licenses.sh

- parse previously generated simple file

According to this post : https://medium.com/@fokusman/the-easiest-way-to-check-all-your-npm-dependency-licenses-753075ef1d9d

It is possible to get a summarized count of licenses

> license-checker --summary

├─ MIT: 949
├─ ISC: 115
├─ BSD-2-Clause: 24
├─ CC0-1.0: 23
├─ BSD-3-Clause: 18
├─ Apache-2.0: 18
├─ CC-BY-4.0: 2
├─ BSD*: 2

Finally you can check if this file contains a specific string and throw an error

---

Mixture (nodejs commands in git lab ci)

If you can execute nodejs commands in your git lab ci, you can invoke validate_licenses.js directly if exists:

test:
    stage: test
    script:
        - echo 'starting licenses validation'
        - node validate_licenses.js

---

References

• https://stackoverflow.com/a/32634749/3957754
• https://github.com/davglass/license-checker/blob/de6e9a42513aa38a58efc6b202ee5281ed61f486/tests/test.js#L233
• https://github.com/davglass/license-checker/blob/de6e9a42513aa38a58efc6b202ee5281ed61f486/tests/test.js#L215
• onlyAllow [list] fail (exit with code 1) on the first occurrence of the licenses not in the semicolon-seperated list

Answer 2

I think this is because of the folder didn't exist till the time you were checking json file in it.

What my recommendation is provide the before script for running the build and then search for the json file in the folder like you are doing in your step definition.