Microsoft Dynamics 365 Test Automation for one of the UK’s leading housing associations

Background

One of the UK’s largest Housing Associations decided to implement Dynamics 365 to manage their business. The implementation was riddled with delays due to issues in the functional implementation and integration with other systems. The customer ended up unexpectedly consuming 70% of the program budget within the first phase alone.

Testhouse was then brought in to investigate their current testing practice and processes to help reduce the overall project’s Cost of Quality. Our 10-vector assessment quickly identified the gaps in the current testing process and automated the regression tests through a series of shift-left practices, coupled with data validations. We were then able to bring the project back on track and timescale within the next 2 phases.

Gaps Identified

• Absence of a consistent approach for testing
• Long cycles of testing due to extensive manual regression packs- 1 cycle took 4 weeks to test 500 test cases
• Limited maintenance of the regression suite leading to a cumbersome set of tests- Exponential test growth rate of 30% with every release
• Testing mindset and functional knowledge/experience of D365 QA was missing
• Automation was UI focused and limited to a few smoke tests- less than 5% automation coverage
• Low E2E test coverage due to lack of integrated test environments and E2E test approach

Solution

• Leverage our 10-vector framework to standardize test assets and processes at an enterprise level 
• Implemented Enterprise level Risk-Based Testing and Regression maintenance strategy
• Optimised regression pack using our 3 -pronged framework
• Automated the optimized regression pack and integrated the same with the daily build to run overnight via the existing CI/CD pipeline using Jenkins
• Established a client university with process maps for the E2E D365 workflows
• Implemented headless automation to identify defects early in the lifecycle
• Trained and upskilled existing personnel in functional and automated script execution
• Architected the continuous testing approach with integrations between the ALM tool, automation framework, and deployment tools

Key Benefits After Phase 2

• Standardized processes/templates/practices helped reduce defects due to ambiguous requirements by 63%
• Optimised regression packs reduced the overall test cycle time (500 scripts were cut down to 290 E2E workflows, 109 duplicates removed!)
• Automation coverage of regression pack increased to 85%
• Cost of Quality reduced (by approx. £25,000 per release) with early identification of defects through service layer testing
• Defect leakage to production brought down to 2% and leakage between other pre-prod phases <10 %
• Successful delivery of Phase-3 & technical debt of Phase 1 and 2 on the initially specified deadline
• Customer ended up with a Self-sufficient and motivated team of automation engineers
• Continuous integration and continuous testing increased speed to market from once a quarter to Once a month.