Create a Lambda function to fetch and store data
The first step is to create a Lambda function that periodically fetches data from an external website and stores it in a serverless database. One way to achieve this is to set up a CloudWatch event to trigger the Lambda function on a regular schedule.
Here's an example of a Lambda function written in Node.js that fetches data from an external website using the request package and stores it in a DynamoDB table:
const AWS = require('aws-sdk');
const request = require('request');
const dynamo = new AWS.DynamoDB();
exports.handler = async (event) => {
// Fetch data from external website
const response = await request.get('https://example.com/data');
const data = JSON.parse(response.body);
// Store data in DynamoDB table
const params = {
Item: {
"id": { S: data.id },
"data": { S: data.data }
},
TableName: "your-table-name"
};
await dynamo.putItem(params).promise();
return {
statusCode: 200,
body: JSON.stringify({
message: 'Data stored in DynamoDB',
}),
};
};
This function uses the request package to fetch data from an external website, parses it as JSON, and stores it in a DynamoDB table using the AWS.DynamoDB client. The name of the table and the structure of the data being stored will depend on the external website you are fetching data from.
Create a Lambda function to retrieve data
The next step is to create a Lambda function that retrieves data from the serverless database and makes it accessible through the API Gateway. This function should be triggered when a user makes a request to a specific endpoint, such as /data.
Here's an example of a Lambda function written in Node.js that retrieves data from a DynamoDB table and returns it to the user:
const AWS = require('aws-sdk');
const dynamo = new AWS.DynamoDB();
exports.handler = async (event) => {
// Retrieve data from DynamoDB table
const params = {
TableName: "your-table-name"
};
const data = await dynamo.scan(params).promise();
// Return data to user
return {
statusCode: 200,
body: JSON.stringify(data),
};
};
This function retrieves data from the DynamoDB table by scanning the table and returns it to the user as a JSON string.
Create a serverless database
We will use DynamoDB as the Serverless database and use terraform to create the DynamoDB table for storing the data.
resource "aws_dynamodb_table" "example" {
name = "your-table-name"
billing_mode = "PAY_PER_REQUEST"
hash_key = "id"
attribute {
name = "id"
type = "S"
}
attribute {
name = "data"
type = "S"
}
}
Integrate with API Gateway
We will need to integrate the Lambda function created on step 2 to the API Gateway so that it is triggered when a user makes a request to a specific endpoint. This can be done by creating a new resource and method in the API Gateway and linking it to the Lambda function.
Here's an example of how to create a new resource and method in the API Gateway using Terraform:
resource "aws_api_gateway_resource" "example" {
rest_api_id = aws_api_gateway_rest_api.api_name.id
parent_id = aws_api_gateway_rest_api.api_name.root_resource_id
path_part = "data"
}
resource "aws_api_gateway_method" "example" {
rest_api_id = aws_api_gateway_rest_api.api_name.id
resource_id = aws_api_gateway_resource.example.id
http_method = "GET"
authorization = "NONE"
request_parameters = {
"method.request.header.Authorization": true
}
integration {
type = "AWS_PROXY"
uri = aws_lambda_function.data_lambda.invoke_arn
http_method = "POST"
request_templates = {
"application/json" = "{\"statusCode\": 200}"
}
}
}
This Terraform code creates a new resource and method in the API Gateway, with the path /data and linked to the Lambda function created on step 2. This will trigger the Lambda function when a user makes a GET request to the /data endpoint and return the data stored in the DynamoDB table.
Verify the Deployment
To verify that everything is set up correctly, you can test the API Gateway endpoint using a tool like Postman and check the logs of both Lambda functions and the DynamoDB table to ensure that the data is being fetched and stored correctly and accessible when the user is logged in.
Wrap Up!
Now by creating these two Lambda functions and a serverless database to our setup, you can fetch and store data from an external website or another service and make it accessible to a our user through an API Gateway using what we learned from other posts we could use the logic to make sure our users are logged in with an identity as well!