In this article I am sharing the full and easy process of creating a AWS lambda function using python. So follow step by step process for creating AWS lambda using python 3.
> fill basic information like name, runtime envoironment, permission
In my case I used s3
-> configure triggers basics
-> select s3 Bucket [bucket name on that you want run your trigger]
-> event type {like bucket PUT}
-> prefix
-> suffix [like - .csv, .jpg]
Note:- you can either choose prefix or suffix.
In my case I had to produced csv name as massage that was automatically uploaded in s3 bucket by python program for athena query result. So I produced massage by kafka producer on s3 put event trigger.
For setting up kafka producer on lambda trigger I did below steps..
#Install kafka in your local machine and create .zip of whole package.
Follow below steps:-
#Now you have to create a file for produced massage on trigger as below -
Lets create a file first with name lambda_producer.py
#Now Lets setup thin within Aws lambda:-
-> choose "code entry type" and select the option to upload .zip
-> upload your zip file and click to save
-> your lambda code has been uploaded.
-> in my case I uploaded kafka configuration zip folder
#You have to also change lambda handler function too. as you are using lambda_handler function within lambda_producer.py file so you have to fill "lambda_producer.lambda_handler" within the above input box [shows in right red line above image].
#Now click on save button.
You done the all steps.
********************** process to create lambda function *****************
> fill basic information like name, runtime envoironment, permission
*******************Now add Trigger from the list in left side****************
In my case I used s3
-> configure triggers basics
-> select s3 Bucket [bucket name on that you want run your trigger]
-> event type {like bucket PUT}
-> prefix
-> suffix [like - .csv, .jpg]
Note:- you can either choose prefix or suffix.
*********************** upload your lambda function code ****************
In my case I had to produced csv name as massage that was automatically uploaded in s3 bucket by python program for athena query result. So I produced massage by kafka producer on s3 put event trigger.
For setting up kafka producer on lambda trigger I did below steps..
#Install kafka in your local machine and create .zip of whole package.
Follow below steps:-
"""
# https://pypi.org/project/kafka-python/
# pip install kafka-python -t <FOLDER PATH>
# pip install requests -t <FOLDER PATH>
# README:
* Please Install the Kafka Python & Request Libs in the same folder for it to work
in AWS Lambda.
* When Uploading on AWS Lambda, zip the entire folder.
If you are uploading a zip file. Make sure that you are zipping the contents
of the directory and not the directory itself.
Else you will get an Error: aws lambda unable to import module
zip -r ../build_dmp_dashboard_from_athena.zip *
"""
#Now you have to create a file for produced massage on trigger as below -
Lets create a file first with name lambda_producer.py
from __future__ import print_function
import sys, time
import json
import requests
from kafka import KafkaProducer
########################### Configurations #############################################
email_api = 'https://api.phpmypassion.com/api/send-mail'
email_to = 'phpmypassion@gmail.com'
kafka_server_cluster = ['cluster1', 'cluster2', 'cluster3']
topic = "athena_query_result"
"""
Send Email
"""
def sendmail(message):
msg = ("{}:{}".format("lambda_producer", message))
data = {
'toEmail': email_to,
'mailSubject': msg,
'mailBody': msg
}
return requests.post(email_api, data)
"""
Producer Connection
"""
exception_counter = 0
for x in range(3):
try:
producer = KafkaProducer(bootstrap_servers=kafka_server_cluster)
break
except Exception as e:
exception_counter = exception_counter + 1
print(e)
print("Error in Connection to kafka cluster: {0}".format(kafka_server_cluster))
time.sleep(2) # 2 Sec
if (exception_counter >= 2):
message = ('Error: Looks like connection to kafka cluster failed on lambda producer: {0}'.format(kafka_server_cluster))
print(message)
response = fnc_sendmail(message)
print("Email Sent Response: {0}".format(response))
print("Program exiting")
sys.exit(2)
"""
Lambda Handler. Received the trigger from the Code Commit Repositoy of AWS and Produces
a message on kafka Queue
"""
def lambda_handler(event, context):
print("Received event : " + json.dumps(event, indent=2))
# Parse the S3 Trigger and Get the CSV Path & File from the "event"
key_file_name = (event['Records'][0]['s3']['object']['key'])
print("We have a new file. file name & path : ", key_file_name)
try:
producer_ret_val = producer.send(topic, key_file_name.encode()) # If Topic Does not Exist then it get created automatically by python Producer
record_metadata = producer_ret_val.get(timeout=10)
print("sent event to Kafka! topic {} partition {} offset {}".format(record_metadata.topic, record_metadata.partition, record_metadata.offset))
except Exception as e:
print(e)
print("Some Error in lambda_producer")
raise e
#Now Lets setup thin within Aws lambda:-
-> choose "code entry type" and select the option to upload .zip
-> your lambda code has been uploaded.
-> in my case I uploaded kafka configuration zip folder
#You have to also change lambda handler function too. as you are using lambda_handler function within lambda_producer.py file so you have to fill "lambda_producer.lambda_handler" within the above input box [shows in right red line above image].
#Now click on save button.
You done the all steps.
Note: Only a member of this blog may post a comment.