Most automated QA flow formulas are based on hard-drive based technologies. However, many companies, including most startups, have been moving over to a serverless environment.
So how does automated QA work in a new serverless environment?
In any startup environment, there is a tradeoff between launching your code now and launching great code. Being able to quickly test separate elements and then move on to the next phase is imperative.
Common wisdom implies that testing in a serverless environment is more difficult than testing on servers; in reality, the serverless system can make testing easier and more efficient – if you set it up correctly.
Testing Code on a Serverless System
The key to quality assurance on a serverless system is to understand your starting point when using the framework.
Older testing processes encouraged coders to record or mimic each interaction within the application. You could then play back the interaction to ensure the application returned the same results. This resulted in a slow testing process that increased build times. These tests were also brittle and every change to the system could break previous tests.
Serverless approaches allow you to skip some of the steps used in monolithic and microservice approaches.
As an example, AWS has pre-tested many of its services. Thus, unit testing can often also test the boundary of each function. When you use external services for most of your interactions, test boundaries often become uncoupled. After successful unit and integration tests, you just need end-to-end tests; when you build functions and have an event driven approach using third party services like AWS, you can limit your tests to the separate functions, and then the system.
While this may make the system testing step somewhat more difficult, you can often use third party tools to test the HTTP endpoints. When everything, including the Lambda, is a service, internal event triggering tests become easier.
Challenges in Serverless Testing
While serverless technology does introduce simplicity to QA testing, it also presents some challenges. Serverless architecture brings together separate distributed services which must be tested independently, and together. The architecture depends on cloud services which can be hard to emulate locally; when you have event-driven asynchronous workflows, it can be difficult to emulate at all.
To deal with these challenges, you should write your business logic separately from AWS or your FAAS provider. In order to keep it independently testable, keep it provider-independent and reusable.
Keeping your business logic separate also allows you to write individual unit tests so that you can ensure they work properly. In addition, write your integration tests with the intent to verify that your integrations with all service providers work correctly.
Put the code responsible for setting up dependencies injecting them, calling business logic functions, and interacting with your FAAS in separate files, so modifying one does not affect all. Then, unit tests can cover each class while you can setup integration tests to invoke the function.
The Serverless Infrastructure and QA Testing
You do not manage the infrastructure when you use serverless apps. However, you can run some parts of your app locally using tools like AWS SAM CLI for AWS Lambda Apps, Azure Functions Core Tools for Azure functions, and other third-party tools like localstack. While these tools do have limitations, they can simulate serverless functions.
Local testing can provide quick validation, but to make sure your serverless app works as intended, you need to write tests. For instance, Node.js testing tools or the Galil Testing Automation framework work smoothly in a serverless environment.
Of course, Mike Cohn’s Testing Pyramid has defined QA for a generation. It tells you to run a bunch of cheap unit tests, proceed to a few integration tests which are costlier to run, and conclude with a handful of expensive UI tests. While Cohn’s guidance is still helpful, it needs to be tweaked for the serverless environment.
You will still run a lot of unit tests. But integration tests also become very important in this scenario because you are running more integrations. They are also much cheaper to run in a serverless environment, so you will have the resources to do more runs. Similarly, the UI tests are cheaper and faster to run because parallelization is less expensive.
In order to write serverless functions, you need to consider the risks unique to this environment. For instance, the configuration can be messy if you don’t have access rights or if your database and tables are not correct. If you don’t parse incoming requests correctly, you can also run into technical workflow problems. You need to follow all of the business logic rules you’ve set up for your application. You also need to make sure you are reading the incoming request structure correctly to avoid integration failures.
However, in the serverless environment, you cannot test each of these processes separately. That would be like assembling and disassembling a refrigerator just to test a screw. Instead, when you apply hexagonal architecture to your serverless functions, you can test things in blocks.
With a hexagonal architecture, you can easily write unit tests because you can use a local or mock adopter to isolate the function business layer. Integration tests can be done through the FileRepository while using tools like Serverless Chrome or Chromeless to run a fast browser in AWS Lambda. Finally, use a CI tool like Claudia.js to run the final tests and deploy the app.
The big stumbling point in serverless QA testing is that sometimes integrations change without notice. This means you need to be using serverless monitoring tools like IOpipe or Dashbird to make sure your code does not break. It is equally important to monitor your front end with tools such as Sentry or Rollbar.
Moving to a serverless environment does not make QA Testing more difficult, but it may mean changing your approach and processes. Once the adjustment is made, however, you’ll quickly find that you have less of a tradeoff between quick deployment and quality code.
Bassel Khoury, VP QA Automation, Galil Software
Resources:
https://martinfowler.com/bliki/TestPyramid.html
https://read.acloud.guru/testing-and-the-serverless-approach-495cef7495ea
https://serverless.com/framework/docs/providers/aws/guide/testing/
https://medium.freecodecamp.org/the-best-ways-to-test-your-serverless-applications-40b88d6ee31e