Customer Overview
Accelerating solution delivery and elevating production quality with an organization-wide SAP QA transformation
The client is a multi-billion dollar high-tech giant with one of the leading portfolios in personal computing systems, printer technology, and 3D printing solutions. It ran a highly customized SAP environment over a legacy tech stack and relied on manual testing to support rollouts. It wanted to bring more agility and efficiency to its systems and decided to transition to a single global S/4 HANA instance. The client wanted to exploit testing automation to achieve higher efficiency, lower risk, and near-zero production defects.
The client engaged Birlasoft to lead its SAP QA transformation. This case study offers insights into the organization's shift from manual to automated testing and how Birlasoft's solution helped save 60-70% of manual effort while achieving higher customer satisfaction.
The Challenge
#1. Suboptimal testing methodology and lack of a standardized testing governance framework
In the absence of a standardized test governance framework for SAP implementations, organizations are exposed to reliability and quality risks during system changes. Such environments are usually characterized by the absence of guidelines and protocols for their testing processes. This results in ad hoc and inconsistent testing practices, which result in varying output quality. In the case of our client, this resulted in overlooked errors and a lack of reliability during system changes and upgrades.
In addition, our client also faced difficulties in testing SAP data models, and BI reports underpinning critical functions like supply chain, finance, and customer service. The lack of an end-to-end, repeatable testing approach negatively impacted the accuracy, reliability, and integrity of core operations.
Therefore, the organization saw the need for a comprehensive test governance framework that would outline the testing objectives and define test environments, roles, responsibilities, and criteria for failed or passed tests. The goal was to minimize the room for errors in production systems, enhance collaboration between teams, ensure consistent baseline quality, and ensure that all SAP system changes were rigorously validated before implementation.
#2. Manual testing contributed to high overheads and low reliability
Manual SAP testing can be highly time-consuming. It not only slows down the organization but also results in the possibility of human errors. Ultimately, inaccurate testing leads to issues trickling into production, which in turn, causes operational disruptions and lowers customer satisfaction. For our client, manual testing also contributed to a lack of collaboration between departments or "towers." This resulted in fragmented testing efforts and incoherent insights into test processes. Business users were overwhelmed with frequent updates and new releases, which lowered organizational productivity.
Our client saw a dire need to break away from manual testing by leveraging testing automation to enhance efficiency and minimize errors. This would free up expensive resources and enable them to focus on more strategic tasks. In conjunction with an effective test governance framework, test automation would lead to smoother SAP operations and improved user experience.
#3. Performance assessment and stress testing for new SAP architecture
Because our client was transitioning from multiple SAP ECC instances to a single global S/4 HANA instance on the AWS cloud, it required a thorough performance assessment to ensure a seamless transition. This was crucial to ensure system responsiveness. Stress testing was necessary to determine if the new architecture was ready to handle the workload.
The performance assessment methodology was to simulate varying workloads to stress-test the SAP ecosystem. This would help understand if the new architecture was ready to handle fluctuations in demand without compromising performance. Stress testing would also offer insights into the stability of the new system and user satisfaction.
#4. Missing testing strategies for supporting growing data analytics adoption
Our client's growing adoption of data analytics highlighted a significant gap in its testing practices. More specifically, the absence of end-to-end Extract, Transform, Load (ETL) testing introduced data inaccuracies and discrepancies. Data quality is crucial in determining data analytics outcomes, as inaccurate data erodes the value of insights, impacts decision quality, and can cause regulatory compliance issues.
For this reason, our client sought to introduce a repeatable ETL testing process that would verify the accuracy and integrity of data as it moved through the analytics pipeline. This included validating data transformation logic, ensuring data consistency, and confirming that the final analytics outputs aligned with business expectations.
To sum it up, the client realized the need for a QA transformation to support its SAP ecosystem. As a result, they engaged Birlasoft to lead the QA transformation and help them exploit test automation to improve the reliability of their SAP deployments.
Birlasoft Solution
Birlasoft delivered a complete QA transformation with a unified functional and component-based testing approach. Leveraging our proprietary Automation Framework TruAut, we were able to accelerate the QA transformation for our client without causing any disruptions to day-to-day operations. ALM, Jeera, and TestRail were implemented to centralize test governance. Testing automation capabilities were built using UFT and TOSCA platforms.
See the highlights of this QA transformation below.
#1. Centralized test strategy and Testing CoE governance model with TestRail
Birlasoft began by defining a comprehensive testing strategy encompassing technical and performance considerations. BI reports, and SAP data models across key programs such as Simplify BI, Nightfall, and Olympus were thoroughly evaluated to devise the testing strategy.
To enhance testing coordination, we established a centralized Testing Center of Excellence (TCoE) governance model using TestRail, a leading test management tool. The governance model enabled System Integration Testing (SIT) and User Acceptance Testing (UAT) management across various business towers; standardized testing practices and provided complete visibility into testing processes. Through streamlined communication and structured governance, we ensured consistent outcomes of testing efforts across different departments and project phases.
Finally, we devised a robust defect tracking and resolution mechanism to expedite the identification, communication, and rectification of issues. This enabled us to minimize the disruptions caused by defects, ensuring that discrepancies were promptly addressed.
#2. Test automation with TOSCA to mitigate manual test challenges
To automate testing for the S/4 ERP system, Birlasoft leveraged the TOSCA platform. Over 400 test automation scripts were developed and executed, targeting critical business transactions in Finance, Order-to-Cash (O2C), and Supply Chain operations. This enabled us to streamline testing processes with guardrails, minimize manual efforts, and reduce the risk of human errors. By automating testing across key business domains, we achieved higher testing accuracy, faster execution, and more reliable SAP operations for our client.
The shift to automated testing also represented a strategic change in testing methodologies. In addition to automating the test processes, we also sought to increase the test coverage, reduce testing cycle times, and improve the overall quality of testing outcomes. This enabled better testing efficiency, accuracy, effectiveness, and velocity.
#3. Agile 24x7 testing and holistic coverage model with real-time dashboards
The dynamic nature of the SAP ecosystem, characterized by frequent updates and changes, requires a 24x7 testing model in place. Such an approach ensures the system evolves into an optimal and responsive state through every release. This optimizes efficiency and maintains a performant deployment. Therefore, we implemented an agile, 24x7 automated testing model to support the client's SAP ecosystem.
A robust test coverage model was established, encompassing functional, regression, and performance testing. It provided holistic coverage across various scenarios and processes, thus enabling the testing of new functionalities and updates on multiple dimensions before they were pushed into production. Birlasoft also built comprehensive dashboards to provide stakeholders with real-time insights into testing progress across all business towers.
#4. Performance testing for the migrated system and end-to-end ETL testing support
Performance testing was conducted during the migration of the S/4 HANA platform to the AWS cloud. This step was crucial to ensure that the target system was performing optimally. The organization aimed to enhance the reliability and effectiveness of BI solutions and SAP platform transitions by systematically assessing technical functionality and performance under various conditions. Finally, we built end-to-end ETL testing support to verify the accuracy and reliability of data as it moved through the ETL processes. Data integration processes were thoroughly validated, enabling reliable analytics outcomes and informed decision-making.
The Impact
With a centralized test governance function and automated testing, the client realized significant benefits in quality and performance enhancements. Savings were achieved through efficiency gains, and ultimately, system reliability elevated the end-user experience.
Quality improvements and reduction in defects
- Achieved 80% improvement in quality, resulting from a significant reduction in production defects.
- Minimal post-production issues, which helped build higher confidence in release decisions.
- 0-1% defect leakage from System Integration Testing (SIT) to UAT stages across all testing cycles.
- Realized higher customer satisfaction (CSAT) due to enhanced reliability and performance.
Efficiency gains through automation
- Created a flexible and repeatable automation framework adaptable to GUI/web-based applications.
- Improved efficiency through streamlined processes, which led to reduced risk and costs.
- Extensively automated test cases with high coverage, resulting in a 75% automation rate.
- Achieved a 60-70% reduction in manual efforts through testing automation with TOSCA.
Robust BI testing and performance improvements
- Released 200+ vetted data models and 300 BI reports into production and conducted thorough BI testing to ensure the reliability and accuracy of critical components.
- Successfully released assets without post-production defects, showcasing the efficacy of the testing strategies.
Successful transition to S/4 HANA with enhanced delivery
- Performance testing enabled the seamless transition of a multi-billion-dollar supply chain from on-prem ECC instances to upscaled S/4 HANA deployment on AWS.
- Accelerated planning and solution delivery by 70%, facilitating faster time-to-market and improved project turnaround.
