Securing Your APIs with Python Jose

Learn how to secure your serverless APIs with AWS Lambda Authorisers and Python Jose. We'll walk you
through the mechanics of OAuth and show you how to build a proof-of-concept with Terraform and Python.

When it comes to building serverless APIs, security is always a top concern. One way to ensure that only authorised users can access your APIs is by using an AWS Lambda Authoriser. An authoriser is a Lambda function that you can use to authenticate and authorise incoming requests to your API Gateway.

we will show you how to use AWS Lambda Authorisers in combination with Python Jose, a library for handling JSON Web Tokens (JWTs), to secure your serverless APIs. We'll start by discussing the basics of OAuth, the standard for authorisation on the web, and then move on to building a proof-of-concept with Terraform and Python.

First Steps!

First, let's take a look at the mechanics of OAuth. OAuth is an open standard for authorization that enables third-party applications to obtain limited access to an HTTP service. It works by allowing users to authenticate and authorize a third-party application to access their resources, without having to share their credentials.

There are three main actors in the OAuth flow: the resource owner (the user), the client (the third-party application), and the resource server (the API). The client sends a request to the resource server for access to the user's resources. The resource server then redirects the user to an authorization server, where the user grants permission for the client to access their resources. Once permission is granted, the resource server sends an access token to the client, which can then be used to access the user's resources.

Now that we understand the mechanics of OAuth, let's take a look at how we can use AWS Lambda Authorisers and Python Jose to secure our serverless APIs.

AWS Lambda Authorizers are Lambda functions that you can use to authenticate and authorize incoming requests to your API Gateway. They can be configured to use various authentication mechanisms, such as JWTs, OAuth tokens, or custom authentication. In this example, we'll use Python Jose to verify JWTs.

Going Deeper

First, you'll need to create a new Lambda function and add it as an authorizer to your API Gateway. Here's an example of a simple Lambda authorizer written in Python:

import json
import jose
def lambda_handler(event, context):
    # Get the JWT from the request
    jwt = event['authorizationToken']
    # Verify the JWT
    try:
        jwt_obj = jose.jwt.decode(jwt, 'secret_key', algorithms=['HS256'])
    except jose.exceptions.JWTError:
        raise Exception('Invalid JWT')
    # Return the user's email address
    return {
        'principalId': jwt_obj['email'],
        'policyDocument': {
            'Version': '2012-10-17',
            'Statement': [
                {
                    'Effect': 'Allow',
                    'Action': 'execute-api:Invoke',
                    'Resource': '*'
                }
            ]
        }
    }

This is a simple example of a Lambda function that can be used as an authorizer for your API Gateway. The function takes an incoming request and extracts the JWT from the authorizationToken field. It then uses the jose library to verify the JWT. If the JWT is valid, the function returns the user's email address as the principal ID. If the JWT is invalid, the function raises an exception.

You'll need to replace 'secret_key' with the secret key that you've used to sign the JWT. The 'email' field should be the field that holds the user's email address in your JWT.

Once you have created your Lambda function, you can add it as an authorizer to your API Gateway. To do this, you'll need to go to the API Gateway console and select your API. Then, select the Authorizers option from the sidebar, and click the Create button. You'll then be prompted to select the Lambda function that you want to use as an authorizer.

With your Lambda Authorizer set up, any incoming requests to your API Gateway will be passed through the Lambda function for authentication and authorization.

Refining the idea

To take this a step further, you can use Terraform to build out a proof-of-concept environment that includes an API Gateway, a Lambda function, and an IAM role for the Lambda function. Here is an example of a Terraform configuration that creates an API Gateway, a Lambda function, and an IAM role for the Lambda function:

resource "aws_api_gateway_rest_api" "example" {
  name = "example"
}
resource "aws_api_gateway_resource" "example" {
  rest_api_id = aws_api_gateway_rest_api.example.id
  parent_id = aws_api_gateway_rest_api.example.root_resource_id
  path_part = "{proxy+}"
}
resource "aws_api_gateway_method" "example" {
  rest_api_id = aws_api_gateway_rest_api.example.id
  resource_id = aws_api_gateway_resource.example.id
  http_method = "ANY"
  authorization = "CUSTOM"
  authorizer_id = aws_api_gateway_authorizer.example.id
}
resource "aws_api_gateway_authorizer" "example" {
  name = "example"
  rest_api_id = aws_api_gateway_rest_api.example.id
  type = "TOKEN"
  identity_source = "method.request.header.Authorization"
  authorizer_uri = aws_lambda_function.example.invoke_arn
}
resource "aws_lambda_function" "example" {
  filename = "lambda_function.zip"
  function_name = "example"
  role = aws_iam_role.example.arn
  handler = "lambda_function.lambda_handler"
  runtime = "python3.8"
}
resource "aws_iam_role" "example" {
  name = "example"
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF
}
resource "aws_iam_role_policy" "example" {
  name = "example"
  role = aws_iam_role.example.id
  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "logs:*",
      "Resource": "arn:aws:logs:*:*:*"
    }
  ]
}
EOF
}

This Terraform configuration creates an API Gateway, a Lambda function, and an IAM role for the Lambda function. The API Gateway is configured to use the Lambda function as an authorizer, and the Lambda function is configured to use the IAM role.

Wrap Up!

AWS Lambda Authorizers provide a powerful way to authenticate and authorize incoming requests to your serverless APIs. By using a Lambda function to authenticate and authorize requests, you can add an extra layer of security to your APIs without having to manage servers or infrastructure.

When building a Lambda Authorizer, it is a good idea to use a JWT library such as Python Jose, to handle the JWT verification. The library provides a simple and secure way to handle JWTs and it's available in multiple languages.

Automating the Build of the ELK Stack with Terraform

The Automatic ELK!

Sounds kind of cool doesn't it? But alas we are not going to be performing any weird experiments today! Looking back at our last article, we can see the basic usage of the ELK stack and how to best integrate it.

As we already know, the ELK stack, consisting of Elasticsearch, Logstash, and Kibana, is a powerful tool for data analysis and visualization. In this post, we'll look at how to automate the build of an ELK stack using Terraform, an open-source infrastructure as code tool.

First, let's take a look at the architecture of our ELK stack. We'll use three EC2 instances, one for each of Elasticsearch, Logstash, and Kibana. These instances will be located in a single VPC, with a security group allowing traffic on the relevant ports (9200 for Elasticsearch, 5044 for Logstash, and 5601 for Kibana).

Next, we'll define our Terraform configuration. We'll start by creating the VPC and security group:

resource "aws_vpc" "elk_vpc" {

  cidr_block = "10.0.0.0/16"

}

resource "aws_security_group" "elk_sg" {

  name        = "elk_sg"

  description = "Security group for ELK stack"

  vpc_id      = aws_vpc.elk_vpc.id

  ingress {

    from_port   = 9200

    to_port     = 9200

    protocol    = "tcp"

    cidr_blocks = ["0.0.0.0/0"]

  }

  ingress {

    from_port   = 5044

    to_port     = 5044

    protocol    = "tcp"

    cidr_blocks = ["0.0.0.0/0"]

  }

  ingress {

    from_port   = 5601

    to_port     = 5601

    protocol    = "tcp"

    cidr_blocks = ["0.0.0.0/0"]

  }

}

Next, we'll define our EC2 instances. We'll use a single EC2 instance type for all three instances, but you could use different instance types if needed. We'll also specify the AMI and user data to install and configure the necessary software:

resource "aws_instance" "elasticsearch" {

  ami           = "ami-12345678"

  instance_type = "t2.micro"

  vpc_security_group_ids = [aws_security_group.elk_sg.id]

  key_name      = "elk_key"

  user_data = <<-EOF

              #!/bin/bash

              sudo yum install -y java-1.8.0

              sudo yum install -y elasticsearch

              sudo systemctl enable elasticsearch

              sudo systemctl start elasticsearch

              EOF

}

resource "aws_instance" "logstash" {

ami = "ami-12345678"

instance_type = "t2.micro"

vpc_security_group_ids = [aws_security_group.elk_sg.id]

key_name = "elk_key"

user_data = <<-EOF

#!/bin/bash

sudo yum install -y java-1.8.0

sudo yum install -y logstash

sudo systemctl enable logstash

sudo systemctl start logstash

EOF

}

resource "aws_instance" "kibana" {

ami = "ami-12345678"

instance_type = "t2.micro"

vpc_security_group_ids = [aws_security_group.elk_sg.id]

key_name = "elk_key"

user_data = <<-EOF

#!/bin/bash

sudo yum install -y java-1.8.0

sudo yum install -y kibana

sudo systemctl enable kibana

sudo systemctl start kibana

EOF

}

Finally, we'll define an output to display the public IP addresses of our EC2 instances:

output "elasticsearch_ip" {

value = aws_instance.elasticsearch.public_ip

}

output "logstash_ip" {

value = aws_instance.logstash.public_ip

}

output "kibana_ip" {

value = aws_instance.kibana.public_ip

}

With this configuration, we can use Terraform to automate the build of our ELK stack. Simply run `terraform apply`, and Terraform will create the necessary resources in AWS. Once the resources have been created, you can access Kibana at the public IP address of the Kibana EC2 instance, using the port specified in the security group (port 5601).

Wrap Up!

In conclusion, using Terraform to automate the build of the ELK stack can save time and effort, and ensure that your stack is consistently configured. Whether you're using the ELK stack for log analysis, application monitoring, or any other purpose, automating the build process with Terraform is a valuable addition to your toolkit.

If you think about it you could take the above and abstract it into a module, it's output could be added to report the endpoint you have created which your Logging instances could then pick up from and configure automatically for the environment they are deploying in to.

Getting Started with Python Logging

On top of our series on the ELK stack we have a small primer with Python to make our code a little more professional and give us something to work with as we test out our Logging stack and play with the various tools we have available to us within the Technology.

The Python logging library is a powerful tool for adding logging to your Python applications. With the logging library, you can log messages to a file, the console, or a remote logging server, and you can control the level of detail that is included in the logs. In this post, we'll look at how to use the logging library, with examples of code and pictures of the terminal.

On top of this we can make our Python code look more professional by removing the print() statements and giving our software a more formalised way of telling the user what is going on.

A very VERY basic Example

First, let's take a look at a simple example of using the logging library to log a message to the console:

import logging

logging.warning("This is a warning message")

When you run this code, you should see the following output in the console:

This is a warning message

Python logging example

As you can see, the logging library includes the log level (in this case, "WARNING") and the log message in the output. You can also specify a logger name, which will be included in the log output:

import logging 

logger = logging.getLogger("logger")
logger.warning("This is a warning message")

This will produce the following output:

This is a warning message

Python logging example with logger name

In addition to logging to the console, you can also log messages to a file. To do this, you'll need to create a logger with a FileHandler:

import logging 

logger = logging.getLogger("file_logger")
logger.setLevel(logging.INFO) 

# create a file handler
handler = logging.FileHandler("log.txt")
handler.setLevel(logging.INFO) 

# create a logging format
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s 

handler.setFormatter(formatter)

add the file handler to the logger:

logger.addHandler(handler)

log some messages:

logger.debug("This is a debug message")
logger.info("This is an info message")
logger.warning("This is a warning message")
logger.error("This is an error message")
logger.critical("This is a critical message")

This code will log messages to the file "log.txt" at the specified log level (in this case, info and above). The log file will contain the log level, logger name, and log message, along with the timestamp.

In addition to logging to the console and to a file, you can also log messages to a remote logging server. To do this, you'll need to create a logger with a SocketHandler:

import logging
import socket  

logger = logging.getLogger("remote_logger")
logger.setLevel(logging.INFO) 

# create a socket handler
handler = logging.SocketHandler("localhost", 514)
handler.setLevel(logging.INFO) 

# create a logging format
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
handler.setFormatter(formatter) 

# add the socket handler to the logger
logger.addHandler(handler) 

# log some messages
logger.debug("This is a debug message")
logger.info("This is an info message")
logger.warning("This is a warning message")
logger.error("This is an error message")
logger.critical("This is a critical message")

The above code, when run will log messages to a remote logging server at the specified host and port (in this case, "localhost" and 514). This could be adapted to point towards our LogStash configuration from previous blog posts. The remote logging server will receive the log level, logger name, and log message, along with the timestamp and this data can be stored in Elastic Search and then viewed with Kibana.

Wrap Up!

There is a Practical example of this in the Python Tutorials Blog talked about previously that deals specifically with this subject, you can have a play for yourself in this Repository.

In conclusion, the Python logging library is a powerful tool for adding logging to your Python applications. Whether you're logging to the console, a file, or a remote logging server, the logging library makes it easy to control the level of detail included in your logs and to customize the log output. With the examples provided in this post, you should be able to get started with the logging library in your own projects. 

[ELK] ElasticSearch LogStash and Kibana Primer!

ELK-in' Around

The ELK stack, consisting of Elasticsearch, Logstash, and Kibana, is a powerful tool for data analysis and visualization. By harnessing the
power of these three open-source technologies, you can easily collect, store, and analyze large volumes of data from a variety of sources.

One of the key benefits of the ELK stack is its ability to handle large amounts of data in real-time. Elasticsearch is a distributed search and analytics engine that can scale horizontally across a large number of servers. This makes it ideal for handling high volumes of data, as it can easily scale up or down as needed.

Logstash is a data processing pipeline that can ingest data from a variety of sources, transform it, and then send it to Elasticsearch for storage. This allows you to easily collect and process data from multiple sources, such as log files, application data, and social media feeds.

Finally, Kibana is a visualization tool that allows you to create interactive dashboards and charts based on the data stored in Elasticsearch. This makes it easy to explore and analyze your data, and to gain valuable insights into your business or organization.

ELK stack architecture

To give you a sense of the power of the ELK stack, let's look at a simple example of how you might use it to analyze log data. First, you would use Logstash to collect log data from your servers and forward it to Elasticsearch for storage. Then, using Kibana, you could create a dashboard that shows the number of log entries over time, grouped by log level (e.g. error, warning, info). This would allow you to quickly identify trends and patterns in your log data, and to alert you to any potential issues.

Configuration is Key

Here's an example of some Logstash configuration that would forward log data to Elasticsearch in a basic context to get you going:

input {
  file {
    path => "/var/log/*.log"
  }
}
filter {
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log_level} %{GREEDYDATA:message}" }
  }
  date {
    match => [ "timestamp", "ISO8601" ]
  }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
  }
} 

GROK

GROK is a powerful tool that w can use to grep or search the log strings to pull them out and enact upon them including ways to tag this data and get it highlighted in the dashboards we will create to make the data more readable.

Wrap Up

In conclusion, the ELK stack is a powerful tool for data analysis and visualization, and is well-suited for handling large volumes of data in real-time. Whether you're looking to analyze log data, application data, or social media feeds, the ELK stack has you covered.

It's strange the number of places I have found this set of tooling, it doesn't matter how big the enterprise, some where there will be a naughty ELK stack stood up somewhere.

In this post, we've introduced the ELK stack and discussed its key components: Elasticsearch, Logstash, and Kibana. We've also looked at a simple example of how you might use the ELK stack to analyze log data, and provided some Logstash configuration to get you started. If you're interested in learning more about the ELK stack, be sure to check out the official documentation and consider giving it a try in your own data analysis projects.

In future posts we are going to be looking to automate the stand up of this set up.

Audio Debugging your Program

It may seem obvious but the number of times I have to teach this in my professional career, it's about time I wrote a handy blog post I can direct people to this page for a refresher is unbelievable, so here it is...

Debugging your Application with Audio!

Sometimes you just want to know that you took the right branch in an if statement, or that the value of a variable is what you expect it to be. Sometimes you don't have the need, time or incentive to fire up the debugger, you just want to dump a few variables to a standard out channel (stdout for example).

echo "here";

This works really well if you're writing a CLI script and it's okay if you're working on some old-fashioned HTML, but when it comes to debugging OAuth issues or (yay!) AJAX requests, it's pretty useless and will get harder as we move towards an API driven future. The pain you go through sometimes to wrangle a response from what you are debugging and hope the data comes back to the browser then find it with something like web inspector, or go grinding through the network activity on a web developer console to find that particular requested document, click to look at the contents and dig out that debug message from the raw source.... So Slow... Not fun!

For me, this method of debugging is really useful since it allows me to debug an issue quickly, of course this is not the perfect method for working on a live service (unless you have to proper precautions in place). However, virtualisation (A blog post on which can be found here) technologies are allowing us to build smaller scale replicas of these systems in less clandestine environments that allow us to reproduce issues quicker and easier - meaning the fix comes quicker and easier!

Where was the audio?

Ok, if you are going to be that picky, now you have primed your execution flow with different logging, or print statements if you are cheating... You can pipe these directly into something like festival or for a Mac, the say command:

$ python execution_flow_test.py | say

You can even change the voice with:

$ python execution_flow_test.py | say -v whisper

See what files you have installed!

$ say -v?

...was that loud enough for you?