0

I'm trying to create a Lambda function that will restart Apache on an arbitrary number of EC2s. There should be a 15 second delay in between each restart operation. This Lambda will be triggered via an ELB and an HTTP request. To avoid the request timing out, I'd like the code to return a response as soon as I get the instance IDs that I want to restart Apache on, but so far, it's only doing one. Is this possible?

My code is below

'use strict'

const AWS = require('aws-sdk')
const ssm = new AWS.SSM()
const ec2 = new AWS.EC2()
const documentName = 'reloadApache'
// Amount of time to wait in between restarting Apache instances
const waitTime = process.env.WAIT_TIME

// Bind prefix to log levels
console.debug = console.log.bind(null, '[DEBUG]')
console.log = console.log.bind(null, '[LOG]')
console.info = console.info.bind(null, '[INFO]')
console.warn = console.warn.bind(null, '[WARN]')
console.error = console.error.bind(null, '[ERROR]')

/*
 * This is intended to communicate with an Amazon ELB. Request / response from
 * https://docs.aws.amazon.com/elasticloadbalancing/latest/application/lambda-functions.html
 *
 * It expects the "event" object to have a member, "path", containing the URL path the function
 * was called from. This path is expected to match either "/apachereload/all" or
 * "/apachereload/<IP>", where <IP> is an IPv4 address in dotted quad (e.g. 10.154.1.1) notation
 */
module.exports.apachereload = async event => {
  const [service, ip] = event.path.split('/').slice(1)

  const ok = {
    isBase64Encoded: false,
    statusCode: 200,
    statusDescription: '200 OK',
    headers: {
      'Set-cookie': 'cookies',
      'Content-Type': 'application/json'
    },
    body: ''
  }

  const badRequest = {
    isBase64Encoded: false,
    statusCode: 400,
    statusDescription: '400 Bad Request',
    headers: {
      'Set-cookie': 'cookies',
      'Content-Type': 'text/html; charset=utf-8'
    },
    body: '<html><head><title>Bad Request</title></head>' +
      '<body><p>Your browser sent a request this server could not understand.</p></body></html>'
  }

  const internalServerError = {
    isBase64Encoded: false,
    statusCode: 500,
    statusDescription: '500 Internal Server Error',
    headers: {
      'Set-cookie': 'cookies',
      'Content-Type': 'text/html; charset=utf-8'
    },
    body: '<html><head><title>It\'s not you, it\'s us</title></head>' +
      '<body><p>Well, this is embarrassing. Looks like there\'s a problem on ' +
      'our end. We\'ll get it sorted ASAP.</p></body></html>'
  }

  const notFound = {
    isBase64Encoded: false,
    statusCode: 404,
    statusDescription: '404 Not Found',
    headers: {
      'Set-cookie': 'cookies',
      'Content-Type': 'text/html; charset=utf-8'
    },
    body: '<html><head><title>Not Found</title></head>' +
      '<body><p>No resources were found matching your query</p></body></html>'
  }

  let response
  console.info('Request ' + event.path)
  // Send 400 back on an unexpected service
  if (service !== 'apachereload') {
    console.info('Rejecting request; invalid service "' + service + '"')
    response = badRequest
  } else if (ip !== 'all' && !ip.match(/^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}/)) {
    console.info('Rejecting request; expected IP address or "all", got "' + ip + '"')
    response = badRequest
  }

  console.log('Event: ' + JSON.stringify(event, null, 2) + '\n')
  console.log('IP: ' + ip + '\n')

  const instanceIds = await getMatchingInstances(ip)

  if (instanceIds.length === 0) {
    response = notFound
    return response
  }

  response = ok
  response.body = JSON.stringify({ instanceIds: instanceIds })
  return Promise.resolve(response).then(reloadInstances(instanceIds))
}

/*
 * Reload Apache on the EC2 instance with the given IP.
 *
 * params:
 * * ip - IP address of the EC2 to reload Apache on. Pass "all" to restart
 *        all EC2s marked as Apache boxen (i.e. with tag "apache" having value "true")
 *
 * returns Promise chaining the AWS EC2 "describe-instances" request and the SSM
 *         "send-command" request
 */
const getMatchingInstances = (ip) => {
  const params = { Filters: [] }
  if (ip === 'all') {
    params.Filters.push({ Name: 'tag:okToRestart', Values: ['true'] })
  } else {
    params.Filters.push({ Name: 'private-ip-address', Values: [ip] })
  }
  console.log('describeInstances params: ' + JSON.stringify(params, null, 2) + '\n')

  /*
   * Retrieve a list of EC2 instance IDs matching the parameters above
   */
  return ec2.describeInstances(params).promise()
    // build a list of instance IDs
    .then((awsResponse) => {
      console.log('describeInstances response: ' + JSON.stringify(awsResponse, null, 2) + '\n')

      const instanceIds = []
      awsResponse.Reservations.forEach((reservation) => {
        reservation.Instances.forEach((instance) => {
          const instanceId = instance.InstanceId
          console.log('Found instance ID ' + instanceId + '\n')
          instanceIds.push(instanceId)
        })
      })
      return instanceIds
    })
}

/*
 * Build a promise chain for each EC2 instance Apache is to be restarted on.
 * For each instance in the list, we will wait process.env.WAIT_TIME ms before going
 * on to the next instance.
 */
const reloadInstances = async instanceIds => {
  return instanceIds.reduce(async (promiseChain, nextInstanceId, i) => {
    await promiseChain
    const waitTime = i < (instanceIds.length - 1) ? process.env.WAIT_TIME : 0
    return restartApacheOnInstance(nextInstanceId)
      .delay(waitTime)
  }, Promise.resolve('INIITAL'))
}

/*
 * restart Apache on one or more EC2 instances. Helper method for apacheReload()
 *
 * params:
 * * instanceIds - array of EC2 instance IDs to send the "restart Apache" command to
 *
 * returns Promise containing AWS SSM "send-command" request for the given instance IDs
 *
 */
const restartApacheOnInstance = async instanceId => {
  console.info('Restart Apache on instance(s) ' + instanceId + '\n')
  const params = {
    DocumentName: documentName,
    InstanceIds: [instanceId],
    TimeoutSeconds: process.env.SSM_SEND_COMMAND_TIMEOUT
  }

  return ssm.sendCommand(params).promise().then(result => {
    console.debug('SSM sendCommand result: ' + JSON.stringify(result, null, 2) + '\n')
    return result
  })
}

/*
 * Returns a promise that waits a given amount of time before resolving
 *
 * Args:
 * * time:  Amount of time, in ms, to wait
 * * value: Optional value that the returned promise will resolve with
 */
// https://stackoverflow.com/questions/39538473/using-settimeout-on-promise-chain
function delay (time, value) {
  console.debug('delay args: ' + JSON.stringify({ time: time, value: value }))
  return new Promise(resolve => {
    console.debug('Wait ' + time + ' ms')
    setTimeout(resolve.bind(null, value), time)
  })
}

/*
 * Attach delay() to all Promises
 */
Promise.prototype.delay = function (time) {
  return this.then(function (value) {
    return delay(time, value)
  })
}
Kit Peters
  • 1,772
  • 1
  • 13
  • 25
  • Can you explain why the code "should return a response as soon as I get the instance IDs" (presumably before you've issues a restart to any of those Apache servers)? – jarmod Feb 12 '20 at 22:06
  • It just feels like good practice? It's intended to be triggered via a REST call, so I figure that it would be good for the thing to return quickly and say, in effect, "OK, request received, I'll get right on it!" – Kit Peters Feb 13 '20 at 15:29

2 Answers2

1

This is a very common problem. The solution is to either use async/await throughout (no callbacks) or to not use async/await at all. Lambda will exit and shut down the process as soon as it reaches the end of your code when using async, but without async it will wait for callbacks to complete.

AWS Lambda Function Handler in Node.js

Jason Wadsworth
  • 5,273
  • 11
  • 22
  • Just so I'm clear, are you saying that I shouldn't use `async / await` in `module.exports.apachereload` (the entry point to my Lambda)? – Kit Peters Feb 12 '20 at 16:38
  • 1
    That's correct. If you need to return a value to the caller you can do so using the `callback` that is passed in. – Jason Wadsworth Feb 12 '20 at 16:40
  • 1
    You don't have to use the Lambda function's callback parameter to return a result to the Lambda service (and hence to the invoker). The handler can return either a value or a promise (which Lambda will await). Examples at https://aws.amazon.com/blogs/compute/node-js-8-10-runtime-now-available-in-aws-lambda/. Most modern Lambda functions won't even declare the callback as a parameter to the handler. – jarmod Feb 12 '20 at 17:37
  • After doing some reading, I see that Lambda is going to freeze the execution context when my function returns, so even if I set it to return immediately and not wait on the event loop to be empty, the additional restart tasks won't get scheduled until the next function invocation. What I really want is to somehow spawn a separate execution thread for the restarts, but I don't know how to do that. – Kit Peters Feb 12 '20 at 20:08
  • 1
    @jarmod is correct, you don't need to use the callback when using promises/async. I was referring to when you are not. – Jason Wadsworth Feb 12 '20 at 20:12
  • 1
    @kitpeters, I'm not sure I fully followed that. Are you saying to want to start a thread on the *next* call into the lambda? – Jason Wadsworth Feb 12 '20 at 20:13
  • No, what I want to do is this: 1) Get EC2 instance IDs 2) Return to the caller 3) Restart apache on the EC2s from #1, 15 sec apart My reading of the Lambda docs suggests to me that I would have to put #3 into a separate execution thread, whether that be through spawning a thread in the existing Lambda (which I don't know if I can do in Node) or by triggering a second Lambda somehow. – Kit Peters Feb 12 '20 at 23:18
  • 1
    One way to do this is for the initial Lambda to enumerate the relevant instance IDs, enqueue them to SQS, and immediately return the list of instance IDs to the caller. Subsequently, SQS will trigger one or more invocations of a 2nd Lambda with one or more of the instance IDs (depending on configured batch size) and those Lambda invocations will initiate the Apache restart for the given instance ID(s). – jarmod Feb 13 '20 at 00:35
1

To achieve that, you have two options (per my knowledge)

1- You need to schedule the task of restarting EC2 to another lambda function. So, the lambda invoked by REST will collect instances' ids and trigger another lambda to restart EC2 (using direct async invoke, SNS, SQS, Step Functions, etc) before responding to the REST caller.

2- (NOT REST) Use a WebSocket API in the API Gateway where you can report the progress to the caller who invoked the lambda by writing to a URL.

Hope that helps.

G. Bahaa
  • 220
  • 3
  • 8