AWS Lambda Scheduled Tasks

Question

Amazon announced AWS Lambda (http://aws.amazon.com/lambda/).

The product description includes:

Scheduled Tasks

AWS Lambda functions can be triggered by external event timers, so functions can be run during regularly scheduled maintenance times or non-peak hours. For example, you can trigger an AWS Lambda function to perform nightly archive cleanups during non-busy hours.

When I read this, I understood I could finally have a way to consistently do "cron-like" tasks. I want to run a specific query everyday at 5PM let's say.

However I do not find this anywhere in the documentation. They only mention triggers on programatical events, or events from other AWS services.

Did I misunderstand? Or can someone point me to the documentation?

Anthony Neace · Accepted Answer · 2017-03-11T03:04:14.443

Native Support for Scheduled Events added October 8, 2015:

As announced in this AWS blog post, scheduling is now supported as an event source type (also called triggers) called "CloudWatch Events - Schedule", and can be expressed as a rate or a cron expression.

Add Scheduled Event to a new lambda

Navigate to the 'Configure triggers' step of creation, and specify the 'CloudWatch Event - Schedule' trigger. Example configuration below:

Add Scheduled Event to an existing lambda

Navigate to the 'Triggers' tab of your lambda, select 'Add Trigger', and specify the 'CloudWatch Event - Schedule' trigger. Example screenshot where I have an existing lambda with an SNS trigger:

Once loaded, the UI to configure this trigger is identical to the screenshot in the above "Add Scheduled Event to a new lambda" section above.

Discussion

For your example case, you'll want to use cron() instead of rate(). Cron expressions in lambda require all fields and are expressed in UTC. So to run a function every day at 5pm (UTC), use the following cron expression:

cron(0 17 * * ? *)

Further Resources

AWS Documentation - Schedule Expressions Using Rate or Cron
AWS Documentation - Run an AWS Lambda Function on a Schedule Using the AWS CLI
AWS Documentation - Tutorial: Using AWS Lambda with Scheduled Events
- AWS has provided a sample "blueprint" that uses a cron expression called lambda-canary that can be selected during function creation from the AWS console.
- This tutorial walks you through configuration of this blueprint.

Notes

The name of this event type has changed from "Scheduled Event" to "CloudWatch Events - Schedule" since this feature was first released.
Prior to the release of this feature, the recommended solution to this issue (per "Getting Started with AWS Lambda" at 42min 50secs) was to use SWF to create a timer, or to create a timer with an external application.
The Lambda UI has been overhauled since the scheduled event blog post came out, and the screenshots within are no longer exact. See my updated screenshots above from 3/10/2017 for latest revisions.

do you know if there is hope for 1 minute precision support? — kaptan, Mar 23 '16 at 00:07
It's a bit of a hack but you can do this for a 1 min frequency https://www.youtube.com/watch?v=FhJxTIq81AU — ingh.am, Apr 05 '16 at 13:33
It does support 1min rate now. http://docs.aws.amazon.com/lambda/latest/dg/tutorial-scheduled-events-schedule-expressions.html — advncd, Nov 13 '16 at 07:29

Shadi · Answer 2 · 2015-07-12T08:13:27.760

Since the time of this post, there seems to have risen another solution: Schedule Recurring AWS Lambda Invocations With The Unreliable Town Clock (UTC) in which the author proposes subscribing to the SNS topic Unreliable Town Clock. I've used neither SWF nor SNS, but it seems to me that the SNS solution is simpler. Here's an excerpt from the article

Unreliable Town Clock (UTC)

The Unreliable Town Clock (UTC) is a new, free, public SNS Topic (Amazon Simple Notification Service) that broadcasts a “chime” message every quarter hour to all subscribers. It can send the chimes to AWS Lambda functions, SQS queues, and email addresses.

You can use the chime attributes to run your code every fifteen minutes, or only run your code once an hour (e.g., when minute == "00") or once a day (e.g., when hour == "00" and minute == "00") or any other series of intervals.

You can even subscribe a function you only want to run only once at a specific time in the future: Have the function ignore all invocations until it’s after the time it wants. When it is time, it can perform its job, then unsubscribe itself from the SNS Topic.

Connecting your code to the Unreliable Town Clock is fast and easy. No application process or account creation is required

Not a valid answer anymore. The Unreliable Town Clock is not available anymore — blueskin, Feb 08 '16 at 21:41

Yves M. · Answer 3 · 2015-10-08T20:09:21.037

NEW SOLUTION: Lambda Scheduled Jobs

Werner Vogel has announced tonight (10/08) at re:Invent that AWS Lambda now has it's own scheduler.

Se the AWS Lambda release note on 2015-10-08 :

You can also set up AWS Lambda to invoke your code on a regular, scheduled basis using the AWS Lambda console. You can specify a fixed rate (number of hours, days, or weeks) or you can specify a cron expression. For an example, see Walkthrough 5: Using Lambda Functions to Process Scheduled Events (Python).

OLD SOLUTION: Scheduling with AWS Data Pipeline

You can use AWS Data Pipeline to schedule a task with a given period. The action can be any command when you configure your Pipeline with the ShellCommandActivity.

You can for example run an AWS CLI command to:

Put a message to SQS
or directly invoke a Lambda function (see invoke)

You can easily create the AWS Data Pipeline scheduled task directly within AWS console (e.g. with an AWS CLI command) :

You can also use the API to define your scheduling:

{
 "pipelineId": "df-0937003356ZJEXAMPLE",
 "pipelineObjects": [
    {
      "id": "Schedule",
      "name": "Schedule",
      "fields": [
        { "key": "startDateTime", "stringValue": "2012-12-12T00:00:00" }, 
        { "key": "type", "stringValue": "Schedule" }, 
        { "key": "period", "stringValue": "1 hour" }, 
        { "key": "endDateTime", "stringValue": "2012-12-21T18:00:00"  }
       ]
     }, {
      "id": "DoSomething",
      "name": "DoSomething",
      "fields": [
        { "key": "type", "stringValue": "ShellCommandActivity" },
        { "key": "command", "stringValue": "echo hello" },
        { "key": "schedule", "refValue": "Schedule" }
      ]
    }
  ]
}

Limits: Minimum scheduling interval is 15 minutes.
Pricing: About $1.00 per month.

score 9 · Answer 4 · answered Jul 14 '15 at 19:07

9

Here is how I do it:

Create Lambda which:
- purges given SQS
- sends there message with delay 10 minutes
- https://gist.github.com/mikeplavsky/5ffe7e33e0d70a248537
Create CloudWatch Alarm for: ApproximateNumberOfMessagesVisible > 0 for 1 minute
Subscribe SNS Topic to the Alarm
Subscribe Lambda to SNS Topic

Now you have a timer with approximately 15 minutes resolution.

Then other Lambda functions are subscribed to SNS Topic and called every 15 minutes.

answered Jul 14 '15 at 19:07

maplpro

346
1
3
11

4

This answer is basically instructions for one mechanism that can implement the Unreliable Town Clock described in [Shadi's Answer](http://stackoverflow.com/a/31354475/18192). Given that the author of UTC warns that it " may disappear without warning at any time" , your approach is probably far superior. – Brian Jul 16 '15 at 15:59
First implementation - 1 message - works couple of days and stops. It looks like periodically message is just not delivered to SQS. Added 2 more messages (3 in sum) - now it works already a week. In CloudWatch I see periodic misses - just 2 messages delivered, still timer ticks fine. Gist has been updated https://gist.github.com/mikeplavsky/5ffe7e33e0d70a248537 – maplpro Jul 24 '15 at 09:08
@Brian As the author of the Unreliable Town Clock (http://townclock.io) I would agree that running your own gives you more control than depending on some stranger (me). In fact, I encourage folks to run their own Unreliable Town Clock and have published the source on GitHub: https://github.com/alestic/alestic-unreliable-town-clock – Eric Hammond Sep 15 '15 at 02:58

score 4 · Answer 5 · answered Jul 30 '15 at 06:59

4

Since it is now easily possible to trigger lambda functions over HTTP (e.g. using GET or curl) a simple solution is to use a managed CRON like easycron: https://www.easycron.com/ to trigger your lambda function into running.

We had the same problem and ended up running a cron service on Google App Engine in python since this allowed for more flexibility and complexity in the CRON job itself.

answered Jul 30 '15 at 06:59

D2TheC

1,903
17
19

Times have changed and Lambda now support cron-type scheduling out the box http://docs.aws.amazon.com/lambda/latest/dg/with-scheduled-events.html – D2TheC Dec 08 '16 at 05:28

score 2 · Answer 6 · answered Aug 03 '19 at 05:09

2

In the Function page, Add trigger, you can add a CloudWatch Events, and make it as a schedule type

answered Aug 03 '19 at 05:09

Vikki

1,317
1
10
18

score 1 · Answer 7 · answered Jun 28 '16 at 14:41

1

You could schedule it with cloudWatch events too. Create rule -> attach target (lambda) and set up cron/rate wise schedule on your rule.

answered Jun 28 '16 at 14:41

johnny

1,772
18
44

score 1 · Answer 8 · answered Feb 13 '17 at 22:33

The web-console way is pretty straightforward. Just create a CloudWatch rule for the lambda and add it in the lambda's Triggers tab.

For those who needs to automate that with aws cli, we can

create the function,
create the rule,
grant the permission,
link rule and function

Create function

aws lambda create-function --function-name ${FUNCTION-NAME} \
--runtime java8 \
--role 'arn:aws:iam::${Account}:role/${ROLE}' \
--handler org.yourCompany.LambdaApp \
--code '{"S3Bucket":"yourBucket", "S3Key": "RC/yourapp.jar"}' \
--description 'check hive connection' \
--timeout 180 \
--memory-size 384 \
--publish \
--vpc-config '{"SubnetIds": ["subnet-1d2e3435", "subnet-0df4547a"], "SecurityGroupIds": ["sg-cb17b1ae", "sg-0e7ae277"]}' \
--environment Variables={springEnv=dev}

Create rules

## create
aws events put-rule --name ${ruleName} \
--schedule-expression 'rate(5 minutes)' \
--state ENABLED \
--description 'check hive connection'

# grant permission to the Rule to allow it to trigger the function
aws lambda add-permission --function-name ${functionName} \
--statement-id 123 \
--action 'lambda:InvokeFunction' \
--principal events.amazonaws.com \
--source-arn arn:aws:events:us-east-1:acc:rule/${ruleName}

# link rule and function
aws events put-targets --rule ${ruleName} \
--targets '[{"Id":"1", "Arn":"arn:aws:lambda:us-east-1:acc:function:RC-checkhive"}]'

score 1 · Answer 9 · answered Mar 08 '19 at 10:29

simple way to run your query in lambda for particular time interval is to set rule for your lambda function. for that after creating lambda function go to cloudwatch>>rules>>schedule. and define cron expression and in the target section select lambda function which you want to trigger.

Traycho Ivanov · Answer 10 · 2020-10-13T14:06:51.777

Run as cron in AWS

An example to setup cloudwatch schedule event trigger for you lambda using cloudformation.


  LambdaSchedule:
    Type: "AWS::Events::Rule"
    Properties:
      Description: A schedule for the Lambda function..
      ScheduleExpression: rate(5 minutes)
      State: ENABLED
      Targets:
        - Arn: !Sub ${LambdaFunction.Arn}
          Id: LambdaSchedule

  LambdaSchedulePermission:
    Type: "AWS::Lambda::Permission"
    Properties:
      Action: 'lambda:InvokeFunction'
      FunctionName: !Sub ${LambdaFunction.Arn}
      Principal: 'events.amazonaws.com'
      SourceArn: !Sub ${LambdaSchedule.Arn}

  LambdaFunction:
    Type: "AWS::Lambda::Function"
    Properties:
      Description: Scheduled lambda to run every 5 minutes
      CodeUri: ./build/package.zip
      Handler: index.lambda_handler
      MemorySize: 128
      Runtime: python3.6

score 0 · Answer 11 · edited Jan 30 '17 at 02:27

0

While creating the lambda function create trigger "CloudWatch Events - Schedule"

Now you can either use AWS presets in schedule expression like rate = 15 min or you can use a cron expression.

enter image description here

For your requirement the Cron Schedule is "0 0 17 1/1 * ? *"

edited Jan 30 '17 at 02:27

Donald Duck

6,488
18
59
79

answered Jan 29 '17 at 18:25

AMS

191
1
17

Nehal Mehta · Answer 12 · 2015-08-14T16:04:29.340

Diksha is AWS Lambda Scheduler based on AWS SWF Trigger as recommended by AWS Team. One can schedule jobs using cron expressions and can also specify how many time you want to run, when to start or when to end. You can view status as well as history of scheduled jobs. Security is managed by AWS policies.

Once you set up diksha engine, you can schedule functions using cron expression in following way:

java -jar diksha-client-0.0.1.jar -lcfg cf1 -cj "jobName|functionName|context|0 0-59 * * * *|10"

In this job job will run every minute for 10 times. AWS SWF will trigger function by itself.

Details: https://github.com/milindparikh/diksha

Disclaimer: I am contributor to the project.