How To Implement Security Testing In Your DevOps Workflow

‍DevOps and security don’t have to be mutually exclusive. In fact, DevOps principles can help you streamline your security testing processes and catch any potential vulnerabilities before they go live. But only if you understand how to integrate security testing into your CI/CD workflow. Otherwise, you risk red tape slowing down the progress of your app until it’s fixed and ready for release again. Security testing is a necessary part of any software development process. You can’t release code that contains bugs or security holes until customers find out about them, unless you want to be the laughing stock of the entire internet. However, integrating security testing into your CI/CD workflow isn’t straightforward. Luckily, we have just the article for you if you need some pointers on how to implement security testing in your DevOps workflow.

What is DevOps and why is it so important?

DevOps is more than just a buzzword. It’s a methodology for integrating software development, operations, and quality assurance into one streamlined workflow. Essentially, it’s a way to make sure that all processes within a software company are working together seamlessly and efficiently. That means that no team member should be working in silos. Operations and developers should be working closely together to make sure that code is secure, scalable, and effective. Similarly, QA has to work closely with developers to prevent security testing issues. No one wants a bad build on their hands when they could have prevented it with a little extra testing!

The importance of integrating security testing

All too often, security testing is treated as an afterthought. Your developers write code, throw it over the fence to QA, and then somehow magically end up with a secure product in the end. Meanwhile, your QA team is frantically trying to test for all potential vulnerabilities and reporting bugs as fast as they can. This is a recipe for disaster. It’s difficult to test for security issues while also making sure that your app works as expected. QA teams are already trying to do too much with too few resources, and security testing is usually the first thing to get cut. Integrating security testing into your CI/CD workflow makes it easier to prioritise security testing. It also gives everyone involved in the process a heads up about potential issues, so they can be fixed before the app gets deployed.

How does CI/CD help with security testing?

Integrating security testing into your CI/CD workflow makes it easier to prioritise security testing. It also gives everyone involved in the process a heads up about potential issues, so they can be fixed before the app gets deployed. If your CI/CD workflow includes automated tests, then security testing becomes easier. You can just write a test that looks for common vulnerabilities, like SQL injections, and then fail the build if there are any issues. Similarly, a CI/CD workflow makes it easy to deploy a new version of your app every time a security test fails. That way, if you find an issue, you can fix it, test it, and then deploy a new version of the app with the fix.

Security testing tools you can use in your DevOps workflow

Now that we’ve covered the basics of integrating security testing into your CI/CD workflow, let’s talk about some of the security testing tools that you can use to do the job. - OWASP ZAP - ZAP or the Zed Attack Proxy is an open source security scanner that you can use to look for common vulnerabilities. It’s not limited to just web apps, either. You can use it to test for security issues with IoT devices, mobile apps, and more. - Selenium - Selenium is a generic automated tool that allows you to test for all kinds of issues, including security. You can use it to create tests to look for things like broken links, incorrect content, and other issues that could impact the user experience. - SSL/TLS Test - SSL/TLS Test is a free online tool that allows you to test the strength of your SSL/TLS settings. It’s a good idea to do this regularly to make sure that your SSL/TLS settings are strong enough to keep you safe from cybercriminals.

Wrap Up!

Security testing is an essential part of any CI/CD workflow. However, it’s difficult to implement if you don’t know how to integrate it into your process. Fortunately, it’s easier than ever to streamline the process, thanks to open source tools like ZAP and Selenium.

Test ALL the things!

Testing your code base is important for ensuring that it does what it's supposed to do. It can also help you identify any bugs before they hit production and save you from spending time on trying to figure out why something isn't working correctly. In this post, we will look at the different types of testing that exist and how each one works so that you can choose which one best suits your needs!

The Types of Testing

There are so many ways to test a code base or a running application created from the running code base. In security terms we have SAST and DAST to represent the difference between the instantiated codebase vs the static code un-run on the screen before us and now even IAST!

Acceptance Testing 

In short, Acceptance Tests allow a team to ensure that the software meets the specifications of the customer. Acceptance testing involves an independent tester performing a scenario-driven evaluation of the software. It is performed by the customer or a representative acting on behalf of the customer.

Acceptance test cases can be used to verify that the software meets all requirements defined in user stories and requirements specifications.

Smoke Testing 

Smoke Tests are used to determine whether a build is stable and therefore ready for further testing.

Smoke testing is a quick way to test whether a build is stable; it's also a good way to find and fix problems in your code base early on, when they're easy to fix. In most cases we turn it on and the off again to see if it behaves correctly (like Power On Self Test/POST when you turn your computer on and hear that beep... or used to when I was a kid!), did it power on, or did it go Boom!? 

Integration Testing

 

Integration Testing is a testing phase in which individual software units are combined and tested as groups in multiple ways. It follows unit testing and precedes system testing.

Integration testing is a testing phase in which individual software units are combined and tested as groups in multiple ways. It follows unit testing and precedes system testing.

The purpose of integration testing is to verify that components work together properly, not that the system works as a whole. Integration tests are designed to test architectural and behavioural aspects of the application or system being tested.

Regression Testing

Regression testing is a technique used to identify any changes to computer programs that might cause previously working features or functions to stop working properly after new code has been introduced. It is a type of nonfunctional testing and can be performed manually, or automatically by computer systems. Regression tests are often used to detect software bugs and ensure that fixes do not introduce new problems in the software under test. They may also be performed with the intention of ensuring that new product versions operate correctly as intended (in some cases, regressions may occur due to changes in product requirements).

Soak Testing

Soak Testing, also known as endurance test, is a type of nonfunctional testing used to determine how a system behaves over an extended period of time when running under peak load conditions. It checks whether the system performs well even after prolonged use under normal workloads by evaluating the response times and error rate during the normal operation stage of stress testing.

Soak Testing is a technique used to identify any changes to computer programs that might cause previously working features or functions to stop working properly after new code has been introduced.

Unit Testing

Unit Testing, is the type of tests where we take individual components and test them individually to monitor their behaviour and ensure they meet the required standard.

Unit Testing can help developers to debug low level issues quicker by testing the response given back to the overall system by the smaller modular components that build up to create it.

Subject Driven Development

In the industry, on top of the above mentioned tests, there are also a number of testing philosophies that will drive there tests through different mechanisms the below is a run down of the different categories seen and which goal each one seeks to achieve.

As someone who has seen all sides of the DD debate, here are a few things I've learned over time

BDD: Behaviour Driven Development

BDD is an approach that aims to avoid defects in code by first writing code for tests, which are then removed when no longer required.

BDD will aim to avoid defects in code by first writing code for tests, generally in a humanistic language, which are then removed when no longer required. It's a test-first approach that uses an agreed upon set of steps (often written as steps or scenarios) and the tools to execute them. The goal is to make sure developers are writing code that conforms with what the customer wants or needs.

TDD: Test-driven development

TDD involves developers writing tests before they write any new functionality. Developers write tests to define the desired results/outputs, before writing the code that will deliver those results/outputs.

Test-driven development (TDD) involves developers writing tests before they write any new functionality. Developers write tests to define the desired results/outputs, before writing the code that will deliver those results/outputs.

As with any new software process, TDD has its fair share of critics and proponents. Some argue that it slows down the team's ability to get features out; others point out that it forces you to think through problems better so you're forced not only to think about what your goal is but how you'll reach it — which ultimately helps identify logical flaws in your logic and avoid dead ends when possible.

DDD: Data driven development

DDD involves gathering data from your top customers. This data can be used to build better products and make your development process more efficient.

Data driven development (DDD) is a software development process that uses data to drive the development process. DDD uses the data gathered from the wider data sources of your company, which can be used to build better products and make your development process more efficient.

Typically the process starts with gathering customer data from your top customers using a variety of methods including surveys, interviews and other research tools including some nifty coding tools embedded into the website. You then use this information to build products that satisfy their needs and wants most effectively.

The goal of DDD is for you to create high quality software with minimal rework at every phase of its creation. In addition, because it utilises customer input throughout each phase, you'll have much more accurate expectations than those based on conjecture alone.

Security Testing!

Because of the size of the subject of testing on the whole, this post hasn't gone into Security Testing at all, there are so many ways we can bullet proof our code in the Security World, it would further turn this post into an essay. Stay tuned for an upcoming post purely on this subject!

Container Testing!

Another huge subject worth of a post of it's own to scratch the surface of the micro architecture realm! I think a few more blog posts would need to be written on different subjects before we could open this can of worms to look at Service Encapsulation and create controls around what are the required tests to achieve the maximum amount of coverage.

Pipeline ALL the things!

With the rise of automation powering what we do, these chores do not have to be so time consuming on an organisation and can therefore be streamlined into a CI/CD workflow. For example we can create a Jenkins Pipeline that would call the individual jobs expressing each type of test, including security tests along the way. We can have those results vetted and flagged only if there is an issue. If a build has not changed much from it's original (A line in the README.md file) is it then required to involve EVERY busy and highly paid professional to make sure that markdown file doesn't now contain something nasty?

Setting up sensible controls across all of your IT infrastructure is the only way that an enterprise can grow without hindrance.

That was a lot to take in!

...it was! But only scratching the surface, with this post we should now be able to discuss various types of testing available from a high level perspective as we dive down deeper in to more in depth technical documents to come.

Jenkins and PHPUnit

Starting from the Beginning

This blog post is another rehashing from the past, bringing some of the older knowledge forward, a lot of it is still relevant today to see behind the scene of the automation technology we are using and in that, gain understanding on how to improve it for the custom functionality we need in today society with modern enterprise. 

The following Blog was co-authored with my senior developer at the time (Zdenek M - Happy to link out, give me a shout, Buddy!) and has helped a number developers and sysadmins integrate Jenkins into their workflow with a variety of different plugins.

This isn't JenkinsX or Pipelines

No, that is correct! This is a very old school method of dealing with things. However, it does show off how the old Hudson (original fork of Jenkins and core code) can handle its plugin architecture.

The Article:

Continuous Integration is the process where code from a developer's repository is called by an automated system, built and tested against a set of standards that the developer has created in their tests. In this post we look at the steps to create a server that will automate your testing suite - and hopefully make you more productive... I have even tried to emulate the links going to website of that time via the Internet Archieve: Wayback Machine!

While working for a PHP Developer shop, we used to have a number of different tools for continuous integration:

• Xinc – to run PHP unit tests and integration tests
• PhpUnderControl (CruiseControl) – to run functional tests
• PHP Unit Testing framework
• Selenium RC Server – standalone server for launching functional tests in web browser

Using all these different tools made things very confusing, and our sysadmin had to learn all these different technologies to try and keep a CI single system together. Some of the main issues with this configuration were:

• Xinc – could be tricky to configure, it looks like a PHP application but doesn't behave like one, it has poor documentation, and to have Git support you have to migrate to the latest version
  
• PhpUnderControl – did a really good job for us over the years, but the project looks half dead and project XML configuration can be quite tricky
• Selenium Server – to be able to run tests in a web browser (Linux) we installed full X Server, which is not the best solution from resources point of view: it needs at least 2GB of RAM
  

To consolidate these systems and their configuration we opted for Jenkins CI which is a continuation of the old Hudson project, available under Creative Commons License, aimed at making CI as easy as possible to implement. The beauty of Jenkins (apart from its single tool structure that contains all CI elements) is that it is a single tool with a pretty, easy GUI that enables easy configuration of a project so that it can be tested.

Another massive benefit is that Jenkins is one of those brilliant tools (like Git) that is so customisable that you can bend it to work in whatever way you need. In this post we shall be looking at how Loft implemented Jenkins into our workflow practices.

So, let's get started...

While planning the server, we listed what services we would need from it:

• Git and SVN abilities - for getting the projects out of our repositories
• PHPUnit
• Xdebug
• Selenium Server
• JUnit - for reports
  

(Not part of the plan but added for fun:)

• Clover reports - code coverage

Once we had the plan the next step was the build...

Jenkins installation

First, we needed a server to set up as our Continuous Integration server (henceforth referred to as CI server). This was to be a Linux Box as it is far easier to work with than a Windows box, as well as  more secure and cheaper. A disadvantage is that you can't run tests in Internet Explorer on this box.

Luckily for us, Jenkins has it's own repo which can be installed on almost any distribution of Linux that has it's own package manager (Such as Yum or Apt). For RHEL and similar you can install the repository with:

sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo
sudo rpm --import http://pkg.jenkins-ci.org/redhat/jenkins-ci.org.key
sudo yum install jenkins
         

And for Debian/Ubuntu:

wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
sudo sh -c 'echo deb http://pkg.jenkins-ci.org/debian binary/ > /etc/apt/sources.list.d/jenkins.list'
sudo apt-get update
sudo apt-get install jenkins
         

This will do all the hard work of installing Jenkins to the server that we are working on, so all we have to do (as root) is start the service:

/etc/init.d/jenkins start

This will start Jenkins and the Jetty (Java-based webserver) that it runs on. Like us, you may decide that the Jetty is not fit for your purpose and might look for something like NginX or Apache to handle a proxy pass to the Jetty which will give you some extra security to hide your Jenkins Interface behind.

Most of Jenkins is very configurable from the management GUI. From this web page you can download and install new modules to the server which will allow you to use different tools to work out more information about your code and project. These plugins.... do just that: they plug into the project configuration area, which enables the user to add more build stages to the project and extra tests.

Selenium Server requirements

For functional testing we use Selenium Tests with the Selenium RC server to control requests to a web server which acts as a real user going through the site, clicking on links and filling in forms, all the while taking pictures of problems and assessing that all the correct information is displayed on the page. One problem with these tests is that they require a GUI or some form of desktop to use the web browser. In previous set ups of CI, Loft tended to use a Full Blown Desktop (Gnome or KDE) which made our sysadmin's life trickier.

In this case we decided to install a different type of GUI on to the server: one that could still be viewed over an encrypted VNC connection and give access to the web browser that Selenium is running, but would also be light enough that it would not impact on the CI potential of the server.

To remedy this we used a combination of:

• xvfb - as the X server
• x11vnc - acting as the vnc server with the -shared -forever -localhost options set
• Firefox - the web browser
  

The installation of the above allowed us to configure a display on xvfb which acted as a blank x server. On top of this platform we could call our web browser to run the selenium tests.

Configuring the PHP project

To run tests, you need a few extra plugins, which are easy to install through Jenkins plugin manager:

• Jenkins Mailer Plugin – to be able to send emails
• Git Plugin – to update tests / code
• SeleniumRC plugin – if you want to run Selenium tests
• Clover PHP Plugin – to be able to publish code coverage
• xUnit Plugin – for publishing test results

The project has a few sections which enable you to configure what to run, when to run it, and what to do with results.

1) Update code / Tests

Jenkins has excellent support for VCS – SVN, GIT – you can choose or even combine different repositories to update tested code or tests.

2) Build trigger

You can choose when the build is started. We use the option to Trigger builds remotely for builds triggered by post-commit (receive) hooks from Git.

Here is a sample post-receive hook used in a Git repo:

  #!/bin/sh 
   PROJECT="/var/www/vhosts/project"
   cd $PROJECT || exit
   unset GIT_DIR
   git pull
   #start jenkins
   wget http://jenkins:8080/job/project/build?token=sdsdjsdjksdjds787jk -O /dev/null

Or for functional tests we use the Build periodically option – for example to run tests every 4 hours.

3) Build

In our case, just a simple command to run all unit tests:

phpunit --log-junit results/phpunit/phpunit.xml -c tests/phpunit.xml  

4) Publishing results

Because we export test results to junit format, we can use JUnit (Java Unit test) Publish to publish results:

5) Notify developers

By publishing results, Jenkins knows if the build was successful or not. If not, you can notify developers - or even point a finger at the person who broke the build!

6) Selenium

If you need to start the Selenium server, you can start it for every build or check that server is running:

7) Code coverage

Code coverage is optional, but it can give you very interesting information:

Round Up!

So as you can see - it's an absolute breeze, far easier than in the past. I recommend you give it a try or expand to get in to the practice of CI/CD, it's not just for big businesses these days with most SCMs now offering a CI/CD testing facility dictated by a YAML file in your repository.

With the above you can even see how the beginnings of Jenkins Pipelines began to form with similar requests being handled with the above console, just now in a "Groovy" (Not so groovy!) Jenkinsfile!