Generate files from SSM parameters and include in Serverless Framework lambda bundle

Question

I have some JSON stored in SSM Parameter Store that I would like to use in my Serverless Framework lambda functions (contains some details about infrastructure generated earlier by Terraform). I can make a call to SSM to get the data at runtime, but Parameter Store has very low throughput limits (40tps by default) so I would likely exceed that almost immediately, and even the higher limits are still far too low to be doing this in production.

More generally, I would like to avoid the overhead of calling external services to retrieve this information, as it will be used in a custom lambda authorizer, so I want it to be fast and not rely on any external dependencies where possible.

I was thinking of retrieving the JSON from Parameter Store and baking it in to my lambda bundle when I do a serverless deploy. I'm happy with the tradeoff of having to re-deploy my backend to when they change.

I could use environment variables, but the maximum size of all environment variables is 4kb, so I can't be putting JSON in there.

I'm using the Serverless Webpack Plugin, which I think might hold the key, but I'm a Webpack novice, and not sure where to start!

score 1 · Accepted Answer · answered Apr 02 '21 at 21:52

Maybe there's a nicer way, but I managed to get this to work by writing a custom webpack loader. This answer is specifically for my case where I needed to export details of multiple cognito pools to a single file for use in a custom lambda authorizer, but the pattern should work for any scenario, and isn't necessarily tied to SSM either (you could generate the file using any method as the loader is just plain Javascript). It brought my Lambda execution times down from ~40ms (using SSM) down to ~2ms.

Template file & usage

First I created an example template .json file, with the structure matching the data I have stored in SSM. This could be anywhere, but I put it in generated/cognitoConfig.json. This is useful for documentation and code assist at point-of-use.

{
    "pools": [
        {
            "_note_": "This is just an example. This file gets totally replaced with the real pool config by cognitoConfigLoader on deploy",
            "clientId":"example-client",
            "name":"example",
            "poolArn":"arn:aws:cognito-idp:eu-west-2:account-id:userpool/example-pool",
            "poolEndpoint":"cognito-idp.eu-west-2.amazonaws.com/example-pool",
            "poolId":"example-pool",
            "poolKeysUrl":"https://cognito-idp.eu-west-2.amazonaws.com/example-pool/.well-known/jwks.json",
            "region":"eu-west-2"
        }
    ]
}

This can then be imported and used within the lambda code (ES6). For example:

import * as cognitoData from '../../generated/cognitoConfig.json';

function getPoolConfig(name) {
    const poolConfig = cognitoData.pools.filter(pool => pool.name === name)[0]
}

Webpack loader

I configured a custom webpack loader that runs against this template file:

const path = require('path');

module.exports = {
    //...other webpack config

    module: {
        // These execute from bottom to top
        rules: [
            // ...other rules (e.g. babel)

            // Retrieve cognito pool information from SSM and store
            {
                test: /cognitoConfig\.json$/,
                include: path.resolve(__dirname, "generated"),
                loader: path.resolve(__dirname, "webpack-loaders/cognitoConfigLoader.js"),
            },
        ]
    }
}

I then wrote a webpack loader that searches for all matching SSM parameters, and writes the contents to the JSON file. The serverless webpack plugin provides access to the underlying serverless object, so the current AWS credentials can be accessed.

For bonus points, I also got it to download the signing keys, but I didn't include that here as I don't want to clutter the answer:

const slsw = require("serverless-webpack");
const { SSMClient, GetParametersByPathCommand } = require("@aws-sdk/client-ssm")
const { fromIni } = require("@aws-sdk/credential-provider-ini")

module.exports = function () {
    const callback = this.async()

    buildPoolsJson().then(
        poolsJson => callback(undefined, poolsJson),
        error => callback(error)
    )
}

async function buildPoolsJson() {
    const poolsParameters= await loadPoolsParameters()
    return JSON.stringify({
        pools: poolsParameters
    })
}

async function loadPoolsParameters() {
    const awsProvider = slsw.lib.serverless.service.provider
    const ssmClient = new SSMClient({
        credentials: fromIni({ profile: awsProvider.profile }),
        region: awsProvider.region,
    })

    const poolParamsResponse = await ssmClient.send(new GetParametersByPathCommand({
        Path: "/terraform/cognito-pools",
    }))

    return poolParamsResponse.Parameters.map(parameter => {
        return JSON.parse(parameter.Value);
    })
}

Generate files from SSM parameters and include in Serverless Framework lambda bundle

1 Answers1

Template file & usage

Webpack loader