Note: This blog is about managing product vendors when you need their product customized, but we’ve previously written a blog about ensuring high-quality performance from a third-party custom software development vendor that you may also find helpful!
It makes perfect business sense to buy a software product, rather than build a custom product from scratch. But sometimes, the core product doesn’t quite do everything you want, and you need it customized. This still makes a lot of sense because the core functionality of that purchased product is likely stable (caveat: unless you’re an early adopter of a new release), hundreds of people likely already use it, and the cost is significantly lower than building it on your own.
What people often miss is the realization that if you require customizations to that product, you need to treat those customizations as full custom code. Building a full project around it with a project manager and test team is essential to the success of these customizations, and potentially the key to the success of your business depending on the criticality of this customized solution.
To help you better understand this, let’s set the stage with an example.
Joe’s IT Shop uses a well-known ERP system that integrates with various other platforms. Joe’s IT shop reaches out to their ERP vendor and asks for help in building a custom report that pulls customer data from the CRM system and combines that with order details from the ERP. This report is just one example of a customization. All RICEW objects (reports, interfaces, conversions, enhancements, and workflows) are examples of other types of customizations that you need to manage and test.
Why is it important to test fully?
If your customization is critical to the business, then you’ll want to ensure your data is accurate. In our example, the custom report was being fed into Joe’s Manufacturing Shop’s shareholder report. One small bug in the custom report could have a huge impact on the business as a whole.
So, you’re saying I can’t trust the product company?
No, not exactly. Product vendors are not built the same as professional custom development companies. They’re built to enhance and support an already existing product, whereas a custom development shop has strong architects who know how to implement complex solutions.
So, while the product vendor will do some testing, more often than not, they’ll only write happy-path test cases. (Pro tip – Approx. 80% of software bugs are discovered in negative test cases). If you don’t test yourself or with an independent partner, you’re now trusting the product vendor and their overly simplified test cases. Not only that, but you’re trusting a long path of communication from the initial request to the final product, and we all know how quickly things can get misinterpreted as they get passed along!
What’s Lighthouse’s recommendation for managing customizations?
Like we mentioned above, treating the customization as full custom code means the project needs managed, scheduled, resourced, and tested. We also highly recommend providing your own project manager and test team. Your project manager can ensure the vendor has sufficient schedule in place, that your internal stakeholders are on board, and that you have access to manage any interfaces to your existing systems. And, your test team can ensure there are sufficient negative test cases, as well as end-end-end scenario test cases, to go along with the vendor’s happy-path test cases.
If all of that seems like too much to ask from your team, reach out to us! As an independent testing team, our job is to verify that all those initial requirements you gave the product vendor are met. Take a peek below at an example of how this came to life for one of our customers:
An eCommerce customer of ours was implementing a commercial product that was heavily customized. Our customer trusted their vendor in those customizations, and only hired us to execute the test cases the vendor built. Before our team even began, we notified our customer that the vast majority of the test cases were happy-path testing, with very few negative test cases. However, due to budget and timing constraints, the customer asked us to stay the course.
When the system went live, a bug caused an item on sale to be automatically discounted at double the discount. Our customer lost money (we’re talking the cost of a medium-sized house), and fast!
In subsequent releases of that software, our customer asked us to build new test cases to prevent a similar situation. Due to sufficient testing and treating these customizations like full custom code, each subsequent release has gone smoothly.
After reading all that, you might now be thinking, “Okay all this is great, but I don’t have a ton of time here. How do I know what to do first?” You’re not alone! The easy answer is to reach out to us and we’ll be happy to chat through the details! However, if you’d still like to know more about how to reduce your risk when implementing large customizations, check out our previous blog on risk-based testing
