Running Python Scripts in AWS Lambda Using Zipped Uploading: A Detailed Guide

AWS Lambda allows you to run Python scripts as serverless functions. However, when your scripts depend on external libraries, you need to package both the code and the dependencies into a zip file for uploading. This guide walks you through how to run Python scripts in AWS Lambda using a zipped upload, including handling dependencies, creating IAM roles, and managing best practices.

Step 1: Setting Up the AWS Lambda Function

Start by logging into the AWS Management Console. Follow these steps to create your Lambda function:

1. Navigate to AWS Lambda Console:
   • Go to the AWS Lambda service from the console.
2. Create a new function:
   • Click Create Function.
   • Choose Author from scratch.
   • Enter a function name like my-python-lambda.
   • Select Python 3.x as the runtime (e.g., Python 3.9).
   • Select or create an Execution role. More on roles in step 7.
   • Click Create Function.

Step 2: Writing the Python Code

Let’s write the Python script that you want AWS Lambda to execute. Below is an example Python function that fetches data from a public API:

import json
import requests  # External dependency

def lambda_handler(event, context):
    # Fetch data from a sample API
    response = requests.get('https://jsonplaceholder.typicode.com/posts/1')
    data = response.json()

    # Process the data (you can modify this part as per your needs)
    return {
        'statusCode': 200,
        'body': json.dumps(data)
    }

Explanation:

• lambda_handler(event, context) is the entry point of the AWS Lambda function.
• This example fetches data from an external API (using the requests library), processes the JSON, and returns it as part of the Lambda response.

Step 3: Installing Dependencies Locally

Lambda doesn’t support installing external dependencies directly on its interface, so you’ll have to install them on your local machine and package them together with your Python code.

Steps:

1. Create a new directory (e.g., lambda_package) to house your script and dependencies.

   mkdir lambda_package

2. Move your Python script into this directory.

   mv lambda_function.py lambda_package/

3. Install external libraries into this directory using pip. Ensure you install the dependencies into the lambda_package folder, not your global environment.

   pip install requests -t lambda_package/

This command installs the requests library (and its dependencies) in the lambda_package folder.

Step 4: Zipping the Code and Dependencies

AWS Lambda expects the code to be zipped before uploading. Here’s how you do it:

1. Navigate to the lambda_package directory:

   cd lambda_package

2. Create a zip file containing all the files and dependencies:

   zip -r9 ../lambda_function.zip .

This command zips everything in the lambda_package folder, including the Python script and installed dependencies, into a file named lambda_function.zip.

Important:

• The -r9 option ensures that the zip is recursive and compressed efficiently.
• Do not zip the folder itself, only its contents.

Step 5: Uploading the Zip File to AWS Lambda

Once your zip file is ready, you need to upload it to your Lambda function.

1. Go to your Lambda function’s dashboard on the AWS console.
2. In the Function code section, under the Code source dropdown, choose Upload from and select .zip file.
3. Click “Upload” and choose your lambda_function.zip file.
4. Save the changes by clicking Deploy.

Step 6: Creating and Configuring IAM Permissions

AWS Lambda functions require an execution role (IAM role) to access resources like CloudWatch for logging, or S3, DynamoDB, etc.

Basic Steps:

1. Go to the IAM Console:
   • Navigate to the IAM Roles section.
2. Create a new IAM Role:
   • Click Create role.
   • Choose Lambda as the trusted entity type.
   • Click Next: Permissions.
3. Attach the basic execution role:
   • Select the AWSLambdaBasicExecutionRole policy. This allows Lambda to write logs to CloudWatch.
4. Attach additional permissions (if necessary):
   • If your Lambda function needs to interact with other AWS services like S3 or DynamoDB, attach relevant policies (e.g., AmazonS3FullAccess).
5. Attach the role to your Lambda function:
   • Go back to the Lambda console, open your function, and assign the newly created role under the Execution Role settings.

Step 7: Testing the Lambda Function

You can now test your Lambda function to ensure it works as expected. AWS Lambda provides a way to trigger tests with custom event data.

1. Create a test event:
   • Click the Test button on the Lambda console.
   • Choose Create new test event.
   • Enter a sample JSON event (you can use the following):

     {
     "test": "data"
     }

2. Execute the test:
   • Click Test again. If everything is set up correctly, the function will execute successfully and return the API data.

Step 8: Monitoring and Logs with CloudWatch

To debug or monitor your Lambda function’s performance, you can use AWS CloudWatch. AWS Lambda automatically logs execution results and any errors in CloudWatch.

1. Go to the CloudWatch Console:
   • Navigate to Logs in the CloudWatch console.
   • You should see log streams corresponding to your Lambda function.
2. View detailed logs:
   • These logs provide details such as execution time, memory usage, and any error messages that might occur during execution.

Step 9: Handling Larger Dependencies with Lambda Layers

If your project grows and you have larger dependencies, you can use Lambda Layers to avoid exceeding the 50MB limit for direct code uploads.

Creating a Lambda Layer:

1. Package the dependencies into a zip:
   • Create a new directory for the layer (e.g., python/lib/python3.9/site-packages/).
   • Install the required packages here:

     mkdir -p python/lib/python3.9/site-packages/
     pip install requests -t python/lib/python3.9/site-packages/
     zip -r layer.zip python/

2. Upload the zip as a Layer:
   • Go to the Lambda Layers section in the AWS console.
   • Create a new layer, upload the layer.zip file, and select the appropriate runtime.
3. Attach the layer to your function:
   • In the Lambda function settings, add the layer under the Layers section.

Best Practices

• Use Environment Variables: Store sensitive data (e.g., API keys, database URLs) in environment variables.
• Set Proper Timeouts: Configure timeouts to ensure that Lambda doesn’t keep running for too long (max is 15 minutes).
• Limit Package Size: Use Lambda Layers for large packages or unnecessary dependencies to reduce the size of your zip file.
• Test Locally First: Use tools like AWS SAM (Serverless Application Model) to test your functions locally before deploying.

Conclusion

Running Python scripts in AWS Lambda with zipped uploading is a powerful method to deploy scalable, serverless applications. By following this guide, you can write complex Python functions, include external dependencies, and package them for AWS Lambda. Keep in mind Lambda Layers for larger projects and ensure to properly manage permissions and monitoring with IAM roles and CloudWatch.

With this approach, you can automate tasks, run web scrapers, handle data processing, and much more using Python in the AWS serverless environment.