Showing posts with label boto3. Show all posts
Showing posts with label boto3. Show all posts

Pythonizing our Web App!

Ok, you asked for it! You all wanted an update for the lambda functions to use another language and
luckily, my favourite one! Python!

So let's get started!

Update the Lambda function to fetch and store data

The first step is to update the Lambda function that fetches data from an external website and stores it in a serverless database. Here is an example of the same function written in Python:

import boto3
import requests
dynamo = boto3.client('dynamodb')
def lambda_handler(event, context):
    # Fetch data from external website
    response = requests.get('https://example.com/data')
    data = response.json()
    
    # Store data in DynamoDB table
    item = {
        'id': {'S': data['id']},
        'data': {'S': data['data']}
    }
    dynamo.put_item(TableName='your-table-name', Item=item)
    return {
        'statusCode': 200,
        'body': json.dumps({'message': 'Data stored in DynamoDB'})
    }

This example uses the boto3 package to interact with DynamoDB and the requests package to fetch data from an external website, similar to the previous example. Note that the function uses python json module to parse data instead of JSON.parse() function in node.js

Update the Lambda function to retrieve data

The next step is to update the Lambda function that retrieves data from the serverless database and makes it accessible through the API Gateway. Here is an example of the same function written in Python:


import boto3
import json
dynamo = boto3.client('dynamodb')
def lambda_handler(event, context):
    # Retrieve data from DynamoDB table
    data = dynamo.scan(TableName='your-table-name')
    # Return data to user
    return {
        'statusCode': 200,
        'body': json.dumps(data)
    }

This function uses the boto3 package to interact with DynamoDB and returns the data to the user as a JSON string using json.dumps()

Re-Creating a Lambda Authorizer

A Lambda authorizer is a Lambda function that you build to authorize access to your API Gateway APIs using bearer token authentication, such as OAuth or JWT.

Here's an example of a simple Lambda authorizer function that can be used to authenticate a user using a bearer token. This function will use the Okta library to validate the token and return the appropriate policy allowing or denying access to the API Gateway endpoint.

import json
from okta.framework import OktaAuth
def lambda_handler(event, context):
    auth = OktaAuth(event)
    claims = auth.verify_token()
    if claims:
        auth_response = auth.get_auth_response(claims)
    else:
        auth_response = auth.get_auth_response(None, 'Unauthorized')
    return auth_response

This example uses the okta library to validate the token passed in the Authorization header and return the appropriate policy allowing or denying access to the API Gateway endpoint.

Updating Terraform Configuration

For the terraform configuration to use the python function in the Lambda function resource the runtime attribute needs to be updated from nodejs14.x to python3.8. Additionally, in order to link the newly created Lambda authorizer function to the API Gateway, we need to update the authorization attribute of aws_api_gateway_method resource to CUSTOM and add authorizer_id to it pointing to our newly created Lambda function.

resource "aws_api_gateway_method" "example" {
  rest_api_id = aws_api_gateway_rest_api.api_name.id
  resource_id = aws_api_gateway_resource.example.id
  http_method = "GET"
  authorization = "CUSTOM"
  authorizer_id = aws_lambda_authorizer.authorizer_lambda.id

  request_parameters = {
    "method.request.header.Authorization": true
  }
  integration {
    type = "AWS_PROXY"
    uri = aws_lambda_function.data_lambda.invoke_arn
    http_method = "POST"
    request_templates = {
      "application/json" = "{\"statusCode\": 200}"
    }
  }
}

Even Re-Create a Hello World Lambda Function

To create a simple "Hello World" Lambda function, you can use the following code:

def lambda_handler(event, context):
    message = 'Hello World!'
    return {
        'statusCode': 200,
        'body': json.dumps(message)
    }

This function returns a JSON object containing the message "Hello World!"

Verify the Deployment

To verify that everything is set up correctly, you can test the API Gateway endpoint using a tool like Postman and check the logs of all Lambda functions and the DynamoDB table to ensure that the data is being fetched and stored correctly, accessible when the user is logged in, and that the authorizer is properly handling token validation.

Additionally, you can test the "Hello World" lambda function by invoking it from the AWS Lambda console or by making a request to it via an API Gateway endpoint.

Wrap Up!

There you have it! You wanted it you got it, but by updating these Lambda functions and the Terraform configuration to use Python instead of Node.js, you can continue to use the same functionality as before but with a different programming language.

Despite updating the dependencies and language, the function code and dependencies together, remains ready for deployment with Terraform to create the same product demonstrating how modular we can make these websites!

The difference in building across clouds!

An Unbiased Opinion of ALL Public Cloud Vendors

If I start to repeat myself in day to day I like to come here to write up the result, partly to vent from the frustration of repeating myself but also to redirect and teach others. As an engineer there is a need to "solve a problem" for good and that is what has driven this post.

If I have said it once, I have screamed it a thousand times. There are many key differences between the big three (four?) cloud providers when it comes to the DevOps Lifecycle. Through out my career at Cloudreach I have witnessed many clients with different motivations all wanting to migrate to the cloud, in some cases migrating between clouds as the original choice did not serve it's purpose.

Which Cloud is the Best?

The true answer is that none of them are the best, it all depends on the companies motivation, and much like selecting technology for your company to use, you need to have a look around firstly at the talent at your disposal, what are the engineers comfy with. There is a way to cheat this a little with automation discussed below.

I have seen too many articles (that I wonder if they were paid for on some level) saying "Their" Cloud is the best and another cloud is rubbish. Many of these I feel stem from a lack of understanding with working with the Cloud platform for a prolonged period.

Azure

I am known as a "Microsoft Hater" and I have my reasons (maybe I will go in to in a Future blog) and it is true that I do not think the most of the company. However, I will admit that for companies that are looking to move directly out of a data centre without changing much, Azure is the way forward. It can provide a fast exit while allowing Information Technology departments to maintain their old policies as you can create a Data centre in the Cloud. This should not be confused with "going cloud" where an enterprise will adopt the antive principles of Cloud Technology and even embrace the chaos of virtual workloads, in Azure, there technology stack is not so good... plus all the security issues that have happened in recent years. CosmoDB anyone? 

AWS

AWS is the all-rounder, they were on of the first to appear having taken the Open Stack framework and run with it, they were the first to market and have had to build with raw elements to cope with new and varied workflows as enterprise discovered how they would adopt the Cloud. Because of this the amount of tooling available from AWS is enormous. Each element catering for a different tool set and circumstance as they push cloud native strategies that force their users to break the mould. Of course with this comes a lot of configuration to main and run your account at peak, a lot of companies and users can get annoyed at this as it can be hard to write the automation to provision all of these services and it will be time consuming for a team to manually implement, especially when you bring things like AWS Orgs into the picture for good practice and security. 

AliBaba... effectively AWS

AliBaba is effectively a rip off of AWS but with less services, should you need to operate in China, I would suggest using this provider as they are already set up behind the great firewall. Their toolkit is essentially the same as AWS, to the point I would wonder if they ripped the code from inside AWS some how to market as their own. Even some of the documentation bares a similar feel with interesting translations.

Either way, it is a public Cloud platform and has its own pros and cons like any other platform.

GCP

GCP is probably the newest for the classical offerings of Cloud, if you don't count the years it spent offering GAPS (Google Applications Service). And with that it has been created differently which gives an interesting twist to the classic check list.

I have found that much like working with the Go programming language, using GCP is very much "Do it the Go/Google/GCP way or pick another platform". This can be great as there can often be less to configure for the setup and a small business can achieve maximum efficiency and security pretty simply with the recommended defaults. However, this sadly falls down when an obscure workload that Google has not found before is required, there just isn't the level of customisability as a more mature platform such as AWS.

Short Verdict 

As you can see that no Cloud is actually the best, like most tools, it depends on what you are doing with it which will mandate what you choose to use! 



Automation

Cloud providers typically offer a native infrastructure as code language that exclusively supports the deployment of resources to their cloud platform. For AWS, that offering is a service called AWS CloudFormation. There are other tools on the market, some of which are sometimes described as “cloud agnostic” due to their ability to support the deployment to multiple cloud providers, which we will discuss in further detail later.

However, if you are looking to provision services on multiple cloud platforms, Terraform is your go-to option. While Terraform supports all cloud vendors like AWS, GCP, Azure, and many others, CloudFormation is confined only to AWS. So, in case your environment involves multiple cloud deployments, Cloudformation is not for you. Suppose you are using AWS resources like EC2, S3, etc., you are best advised to stick to Cloudformation or the platform equivalent. 

Wrap Up!

Cloud Computing has revolutionised the world of DevOps. It is not just a buzzword anymore; it is here and it is here to change the way we develop and maintain our applications. There are very few companies who are not using Cloud Technology as part of their set up to speed up their delivery cadence.

In future posts we will explore the different automations of each of these Clouds and see if we can unite them!









 

Happy 10th Birthday Securing the Cloud!

It's my birthday and I'll blog/tag/label what I want to!


So while doing a lot of the prep work to create this site, I began organising my old code bases, ripping the good from them and create illustrations and examples... Hilariously I noted some of the dates of the code I was going over and some of it, over a decade in age!

I thought that deserved a little celebration and for some giggles and anonymised list of shenanigans I have witnessed over the times as I have grown with the Cloud.

Top 10 Funniest Moments!

  1. Clients constantly asking for data to be removed from interesting places: When people hear "Security" they immediately jump to hacker and "can you hack $thing?" or "could you get rid of this photo from $X?" this has lead to some funny things I should not talk about...
  2. Client Celebrations: One too few of these, it's always about what happens after the work is done!
  3.  Working with a remote client to do some Network Gymnastics: A tale of deadlines, fast paced delivery and a broken VPN leading to a need to create a reverse shell from the clients system, to me, to connect to the network as SSH direct to the instance was banned. The things we do for our clients!
  4.  Reverse Engineering and Early Days API: Before we had standards and everyone was building rapidly, reverse engineering how things like the Foreman were working to create wrappers and scripts like YAFCT to interact provided so, so many sleepless and coffee driven evenings, I do miss those times!
  5.  Getting told off for talking about Security Vulnerabilities: ...Probably should have lead with the fact that the exploit involved an toy that is more targeted towards adults... <INSERT EGGPLANT EMOJI HERE>
  6.  BitCoin mining rig found in Production: A security lesson to monitor ALL regions regardless if you are using them or not! Sadly the private keys were invalidated, it would have been worth a lot right now!
  7.  Playing Apache Pong and Gource with one of the busiest sites on the Internet: ....Then watching it drop to nothing as the paywall was implemented!
  8.  Using Facebook links to wind up the Web Developers with XSS: A warning to all developers to sanitise your inputs and not to trust anything, including the URL variable!
  9.  Migrating Successfully to the Cloud so well we brought the original malware along too: A promotion for rebuilding into the cloud instead of straight migration - finding out you have pulled malware that has been living on your DC system to the Cloud is not the best surprise, but a funny experience!
  10. Always Staying in the wrong hotel: If you know the story here, you know about the fun and games. Sadly, this one cannot be discussed!

Wrap up!

I hope you have enjoyed a quick look down some of the fun shenanigans as much as I have over the years, maybe throughout this blog you will find other references to some of the above, but as well, don't be afraid to reach out and get in contact!

...with Python Tutorials for All!

A shameless plug for a set of tutorials I made a few years ago that can be see here.

I wrote these back at a time where I was learning lots of different parts of the Python Programming language and thought it would be a good idea to give this information back to the community along with the shortcuts I have found for myself along the way, they might not work for everyone but they have helped me gain a deeper understanding of the language as it has grown over the past decade.

A lot of these challenges have been created over the years for various meet up groups aimed at helping others up skill with programming as a whole!

One of the fun talking points is that the .ipy notebooks themselves were generated automatically from a small quick writer I created that would create the repository above and even upload these into Google for the class to copy and use. Neat, huh?

So let's get stuck in shall we!

The PythonTutorials Repository

I had initially designed this repository to focus on the varying skill levels that would be attending the different meet ups, trying to rank the challenges and build a foundation of knowledge with each student before moving on to the next, harder topic. There is a point as the harder subjects sub-divide that I did wish I had raised them in series, e.g. Security Tasks vs Plain Coding tasks and a whole section just dedicated to the different ways of making games (which seems to be a subject with an ever increasing interest from new coders to the scene).

Beginners

The section for the beginners started out very simplistically, it aims to get the user up and running with the installation of the Python compiler. Having been to many clients where they were adopting the language for the first time, there will be some people who are not able to do things right from the get go, by writing these guides and expanding upon them as problems arose, it became a self developing Bug Fix manual for the classroom.


InstallingPython
This would focus on the generics of what is happening during the installation of Python, no matter what Operating System you are going to install Python on, there will always be the common factors of what will be installed and what the installer will typically do.

InstallingPythonOnLinux
Does what is says on the tin, the very simplistic instructions for installing Python on *nix systems, as most ipy notebooks are Linux based this can even be demonstrated. It made for some fun reverse engineering of the environment for me while creating!

InstallingPythonOnMac
Despite Mac's being very similar to Linux, having used one for a long time I am always reminded of the following XKCD. I thought it might be wise to get a unified way of installing on Macs for the class to follow to avoid this scenario.

InstallingPythonOnWindows
Always the biggest problem from an installation point of view, for some reason there are more issues when it comes to installing on Windows, this guide hopefully helps get people around the discovered pitfalls and saves you time jumping in!

ExploringPythonObjects
My hidden toolkit of Python built-ins that have taught me how to code over the years, when probing an unknown variable this tool kit has proved invaluable. Another blog goes in to getting this data back out of a highly complex system.

IntroductionToPythonEnvironments
A primer to help a user gain an understanding in to what has been accomplished by installing and a little taste of what is now possible.

ExtendingPythonEnvironments
A look into growing the python environments capability with modules and virtualisation!


LogAllTheThings
To go with the above tutorial on Exploring Python objects, the logging tutorial will help get a user set up to prepare to log across different channels so that no matter where a developer needs to dig information on a random variable from, or test execution flow. They are able to extract this data without having to break the more in depth debugging tools that can be daunting to new users.


Intermediate

The Intermediate level assumes an environment already set up by the student and a keen interest to get stuck in, it was at this point I feel I really fell over with my organisation as there was just so much information to record, so many directions to take the various students in and no idea what each individual might want to learn... So the following mess happened!

Cloud

As the Cloud was where I began properly with Python (I mean actually using the language as a programmer and not just writing simple single use scripts), I thought it might be a good way to show the student what I saw, AWS's API seemed to simple and most programatic so a lot more time was dedicated here.

AWS

Although, the original intention was to expand to exercises across all Clouds, time only allowed me to get these AWS tutorials out the door and into the repository, it would have been good to build a foundation understanding and broaden the students mind with the others and show the differences between their APIs and how to research them, I only got as far as the Boto3 setup before having to resort to suggesting challenges to the students, this was another great use for my Random Value Generator project though.

Boto3Setup

A guide to helping the student set up the required to use boto3 Python module, helping them to ramp up into the cloud and start playing with the AWS APIs as well as a brief tour of all the ways a user can provide their AWS credentials.

Games

Games are always a popular subject, I think it's the reason half of us got in to computers in the first place and no surprise these were the most popular tutorials. The crazy thing was that there are now so many ways to write your game, so many different frameworks that will specialise in the different ways that our product could be consumed at the end of it all. Maybe in time I will be able to expand on these, Kivvy is a fun library I was playing with recently but have not managed to include in this repository.

PyGameIntroduction

Only a small introduction in to the massive world of PyGame alone, this will look at helping a student get setup with the module and any additional installations that are needed and ways to test it has worked.

PyGame

A quick look into PyGame to build on the above and guide a student through a quick build to make something silly and spark ideas of what is possible with the language and the PyGame Framework provided by the Module.

Security

It would be with this security section and the intended split with Red and Blue team operations that I feel my sorting came undone, it would have been better to split these out into different repositories, but that can be for another day, the data is still recorded.

Red Team

Originally, I had planned to put the past 2 parts of the buffer overflow exercise here. However, when they were put to the class a lot of them had troubles and I felt they still needed more education in this subject, pushing them up a difficulty level. Sadly, I didn't get around to replacing these just yet but always on my TODO list!

Blue Team

BlueGate-PythonPoC
At the time of authoring the BlueGate exploit had just become mainstream, there was a lot of panic to get this bug patched and I created an exercise to take the attack to pieces and create a testing tool for users to take back to their organisations and ensure they were covered, or alert their Security Teams.

IncidenceResponse-MargaritaShotgun

Another tool I had been playing with for a Proof of Concept for a client, was the idea of automatically snapshotting remote instances memory in a Cloud Environment, this seemed like the best tool for the job and was a lot of fun to play with, so I created a little tutorial that I hoped to expand with the Cloud Exercises above.

Hard

The Harder section of these tutorials were designed to provide more complex applications, consuming the snippets of code we created in the above tutorials we would dive deeper into the language and even computers as some exercises teach you to use python to control other tools such as debuggers to decompile binaries for exploitation!

Security

Of course the first set of hard challenges I would write would revolve around security, especially more the Offensive side. Although, I recall writing a few other challenges in this vein, I don't think they made the final cut as the code was still plagued with bugs!

Red Team

BufferOverflow-Part1-4
A Small guide helping students to learn the principles of the simple buffer overflow attack, the guides help to set up the binaries as well as provide instruction of how to use python to break into the binary and make it run our code as opposed to it's intended use.

Blue Team

Sadly, I didn't get around to writing any fun challenges for Blue Teams to handle, at least at the moment. I hope to be able to dust this repository off and give it some more love in the future, time is always a limited resource.

Give them a try and see what you think!

All the code can be found here or can be pulled from GitHub with the following:

$ git clone https://github.com/s1l0uk/PythonTutorialsForAll
This will get you the tutorials and you can then upload these to a Jupyter Notebook (even hosted on Google) to get you up and running with the language. If you are using it, give it a star to let me know and I will look to add more in this direction! Happy Hacking!