Automated software testing in a Start-up

The first phase of development

My companions and I founded Paybix  at the beginning of 2022 and immediately wanted to start with the build of our International Payroll platform: Epix.

We quickly found a great development partner and together we ran sessions on the SaaS architecture, combined with high level user flows to trust our ideas to paper.

Already after a few weeks, our technical architect created the first building blocks in our Azure datacenter. A great advantage of Microsoft Azure is the fact that when you register as a start-up for the Microsoft Founder Hub, you benefit from a large free spend of Azure freebies and credits.

Early last summer, our first screens were born. At that time, everything was of course still very basic: a login screen, a skeleton menu bar, an almost empty home screen and some person data. We tried to restrict every feature to the bare minimum to speed up development.    

We needed to learn the combination of components, frameworks and libraries with all the possibilities and limitations. As a result, the first screens tended to change frequently during development and features changed when seeing the exact implementation.

As for the testing part: of course, our developers started from day one with unit tests in the code, testing their own code. But as soon as the initial screens started to have some business value, we wanted to demo the live screens instead of some mock-ups.

So, we then set up an acceptance environment to run stable demos, while development could continue to frequently push new EPIX features to our development version.

‍

A previous experience

In my previous role as CTO in a larger company, we invested a lot in professionally testing our software products. Not only done by the developers, but also by our QA team who automated loads of tests.

This way of working allowed us to deploy our software twice per day for more than 500,000 users without downtime, and without any human intervention. All the code created by the development team was published to a test server, and there all our tests were run.



As soon as all the tests were green, the code went into production, based on a blue/green deployment.

Now, at Paybix, we have immediately introduced this way of working in our end-to-end development methodology, as we want to constantly release new features for our customers. We strongly believe in great user experience, and our approach will allow us to quickly respond to feedback from Epix users.

The big question: When do you start?

Now, when do you start with automated QA?

In theory one could say: from the very first moment a letter of code is written. And that would be right of course!

But when you’re a start-up, and you need to be careful with your spendings, the focus from business must be on ‘functionality’ first.

        
And that’s why we wanted to start with the automation of our Epix testing at the right time:

• not too early because then we would have a lot of rework,
• but before we would need a quality feeling going to acceptance (and production),
• and certainly as soon as we would have our first customer.

Well, we found the right moment.

The trigger to start thinking of automated testing was the fact that after about 4 months of development and about 2 months before going to production, we had a very basic error in our acceptance version, which occurred during a demo for a prospect.

Changing a user setting that was implemented in the early days created an issue, because simultaneously we changed something in the backend for another feature.    
The fact that we missed this very basic feature going to acceptance was our trigger to initiate automated testing.

So, I contacted a senior freelance technical QA person from my network. After a quick introduction with the development team, he set up the environment & pipeline and created tests for almost every screen.      
We decided to use Cypress to build our tests, although we also looked at PlayWright.

The number of test cases is limited but we now already run 40 frontend tests on 4 screen sizes in a pipeline that runs every time we publish to acceptance.

From now on, our technical QA colleague will spend 1 day per 2 weeks on average creating automated tests.

This way, we will add tests for new functionality and perform multi browser testing. Every time we find a bug, we will add an automated test to make sure it will never occur again.

Conclusion

There is no other way than to start with automated testing early in development. Not from day 1 (except unittests), but before you go to production with your first customers. This comes with a cost, but not testing or only manual testing will cost more in the end. And the gap will become too big.

Also releasing fast will help get fast feedback from customers, and this is only possible if you have automated testing.

Automated testing is the only way to be able to release fast and frequently, which surely helps to get immediate feedback from customers so you can keep your finger on the pulse for your software improvements.

One of the major advantages is that if something goes wrong, you just implement an extra automated test, you solve the issue, and you quickly automatically retest your software.

Interested in our international payroll services or our EPIX Saas software to optimise your aggregated insights or unified processing?  Contact us.