Software Testing – Mainframe Testing

Mainframe testing is used to evaluate software, applications, and services built on Mainframe Systems. The major goal of mainframe testing is to ensure the application or service’s dependability, performance, and excellence through verification and validation methodologies, and to determine if it is ready to launch or not. Because CICS screens are custom-built for specific applications, the tester just has to know how to navigate them when performing Mainframe testing. Also, if the code in COBOL, JCL, or other languages is changed, the tester does not have to worry about the emulator being set up on the system.

Here, we will discuss the following points:

1. What is Mainframe?
2. Mainframe Testing Methodologies.
3. Prerequisites for Mainframe Testing.
4. Mainframe Attributes.
5. Steps for Mainframe Testing.
6. Mainframe Testing Procedures To Follow.
7. Types of Mainframe Manual Testing. 
8. Mainframe Automation Testing Tools.
9. Best Practices For Mainframe Testing.
10. Mainframe Testing Challenges and Troubleshooting.
11. Benefits of Mainframe Testing.

What is a Mainframe?

The mainframe is a high-performance, high-speed multi-user computer system. The mainframe machine system is the most secure, scalable, and reliable machine system available. In other words, these systems are utilized for larger-scale computing, which requires a high level of availability and security. Mainframe systems are commonly employed in industries such as retail, insurance, finance, and other essential areas where large amounts of data must be processed several times. One can perform millions of instructions per second [up to 569,632 MIPS] with the help of the following factors:

• Maximum input/output bandwidth: If there are excessive input and output bandwidth, the links between drives and processors have a few choke points.
• Reliability: Mainframes frequently agree to graceful deterioration and service while the system is running.
• Reliable single-thread: Performance is critical for realistic database operations.
• Maximum Input/Output Connectivity: Maximum input/output connectivity indicates that mainframes excel at delivering large disc farms.

Mainframe Testing Methodologies

Some of the most commonly used Mainframe testing commands are as follows:

• SUBMIT: This command is used to submit the background job.
• CANCEL: This command is used to cancel the background job.
• ALLOCATE: This command allocates a dataset.
• COPY: This command is used to copy a dataset.
• RENAME: This command is used to rename the dataset.
• DELETE: This command is used to delete the dataset.
• JOB SCAN: This command is used to fix the JCL with libraries, program files, and so on without implementing it.

Prerequisites For Mainframe Testing

Below are some of the prerequisites of mainframe testing:

• A login ID and password are required to access the application.
• A basic understanding of ISPF commands.
• The file names, file qualifiers, and kinds are all listed.

The following points should be checked before beginning mainframe testing.

1. Job:

• Before performing a job, do a job scan (Command – JOBSCAN) to check for problems.
• The test class should be specified in the CLASS parameter.
• By utilizing the MSGCLASS argument, one can direct the task output to a spool, a JHS, or wherever else one wants.
• Redirect the job’s email to a spool or a test mail ID.
• For initial testing, comment out the FTP steps and point the job to a test server.
• If the job generates an IMR (Incident Management Record), just comment “TESTING PURPOSE” on the job or param card.
• All of the job’s production libraries should be switched to test libraries.
• It is not a good idea to leave the job unattended.
• TIME parameter should be added with a specified time to avoid the job from running in an infinite loop if there is an error.
• Save the job’s output, which includes the spool. XDC can be used to save the spool.

2. File:

• Only make a test file of the required size. When storing data into successive files with the same name, use GDGs (Generation Data Groups – Files with the same name but sequential version numbers– MYLIB.LIB.TEST.G0001V00, MYLIB.LIB.TEST.G0002V00, and so on).
• The files’ DISP (Disposition – defines the procedure for keeping or deleting the dataset following a normal or abnormal step or task termination) parameter should be coded correctly.
• To avoid the job going into HOLD, make sure all of the files utilized for job execution are saved and closed appropriately.
• If you’re using GDGs to test, make sure you’re pointing at the correct version.

3. Database:

• Ensure that no undesired data is inserted, changed, or deleted while running the job or online program.
• Also, make sure you’re testing in the correct DB2 region.

4. Test Case:

• Always check for boundary conditions such as an empty file, the first record being processed, the last record being processed, and so on.
• Include both positive and negative test conditions whenever possible.
• Include test cases to validate if the modules have been utilized correctly if standard procedures are used in the software, such as Checkpoint restart, Abend Modules, Control files, and so on.

5. Test Data:

• Before you start testing, make sure the test data is ready.
• Never make changes to the test region’s data without first informing the user. Other teams may be working with the same data, and their tests may fail.
• Before copying or accessing the production files, sufficient authorization should be obtained.

Mainframe Attributes

The following are the various mainframe attributes:

• Multiprogramming:
  • The multiprogramming feature is a feature that helps us to make the most of the CPU.
  • The computer is running many programs at the same time.
• Time-sharing:
  • Foreground processing refers to time-share processing, whilst Background processing refers to batch job processing. As a result, it is referred to as Interactive Processing since it allows the user to interact directly with the computer.
  • In a time-sharing system, each user has terminal device access to the system.
• Virtual storage:
  • As an extension of physical storage, virtual storage makes use of disc storage.
  • It is a method of efficiently using memory to store and accomplish a large number of operations.
• Spooling:
  • The Spool stands for Simultaneous Peripheral Operations Online, and it’s used to collect a program’s or application’s output.
  • If necessary, the spooled output is sent to output devices such as a printer.
• Batch processing:
  • Batch processing is a technology that allows us to complete any task in pieces called jobs.
  • One can run one or more applications in a specific order depending on the tasks at hand.
  • The job scheduler comes to a conclusion regarding the order in which the jobs are executed.
  • To maximize the average production, jobs are arranged according to their priority and class.
  • With the help of JOB CONTROL LANGUAGE, batch processing provides us with the necessary information (JCL).

Mainframe Attribute

Steps for Mainframe Testing

Mainframe testing can be done in two ways- manually or with the help of automation tools like QTP, REXX, and IBM application performance analyzer. The following are the steps for mainframe testing:

Step 1: Make a Plan

To begin, the business or development team builds test plans based on the Business requirement document, System requirement document, other project documents, and inputs. It also dictates how a particular item or process will be modified during the release cycle. In the meantime, the testing team will collaborate with the development and project management teams to prepare test scenarios and test cases in advance.

Step 2: Make a Schedule

Once the requirement document has been appropriately written, it will be turned over to the development and testing teams. In addition, the testing schedule should be created in line with the precise project delivery plan.

Step 3: Deliverables

After receiving the paper, they will review the deliverables. The deliverables should also be well-defined, with no ambiguity, and meet the scope of the test objectives.

Step 4: Implementation

The implementation should next proceed in accordance with the plan and deliverables. In most cases, the modified requirement in a release will directly affect 15-25% of the application. The remaining 60-75 % of the release will rely on out-of-the-box features such as application and process testing. As a result, there will be a need to test the Mainframe application twice-

• Testing Requirements: The application will be tested for the features or changes specified in the requirement document.
• Testing Integration: This testing activity is just for the purpose of testing. The complete procedure will be put to the test, as well as any other apps that receive or transmit data to the important application.

Step 5: Reporting

The test results will be shared with the development team on a regular basis after that. The testing team should connect with the development team to make fast modifications in crucial instances to maintain consistency.

Mainframe Testing Procedures to Follow

When undertaking mainframe testing, keep the following steps in mind:

Step 1: Smoke Testing

Start with smoke testing to see if the code deployed is in the right test environment. It also ensures that the code is free of important flaws, saving time and effort for testers who would otherwise have to test a bad build.

Step 2:  Testing/System Testing

Following the smoke testing, one round of functionality or system testing will be done to evaluate the functionalities of several models independently and in relation to one another. The sorts of testing that must be performed while implementing System Testing are listed below-

• Batch testing: Conduct batch testing to verify that the test results on the output files and data changes made by the batch job comply with the testing specifications.
• Online testing: Evaluate the mainframe applications’ front-end functionality via online testing. Online testing covers a variety of topics, including user-friendliness, data input validations, look and feel, and screen navigation. Exact entry fields, such as interest on the plan, an insurance plan, and so on, should be tested in the application.
• Online-batch integration testing: On systems with batch processes and online applications, online-batch integration testing can be performed. The online process’s integration features with the backend process will also be tested here. Essentially, this testing verifies the accuracy of the data flow and the interactions between the screens and the backend system. Furthermore, the batch task is utilized to verify data flow and communication across the online screens.
• Database testing: Database testing is performed to ensure that the data stored by transactions meet the system’s requirements. And the databases, which contain data from mainframe applications such as IMS, IDMS, DB2, VSAM/ISAM, Sequential datasets, and GDGs, validated their layout and data storage. The data integrity and other database parameters may also be validated for optimal performance during database testing.

Step 3: System Integration Testing

System integration testing is used to verify the functionality of systems that are related to the system under test. Because it’s vital to test the interface and various types of messages like Job Successful, Job Failed, Database Updated, and so on, it’s run after unit tests. The data flow between modules and apps will also be checked for accuracy. System integration testing is carried out to ensure that the build is ready for deployment. One can execute the following tests during system integration testing-

• Batch Testing
• Online Testing
• Online -Batch Integration Testing

Step 4: Regression Testing

Regression testing is the most crucial part of any testing. Regression testing ensures that batch jobs and online screens cannot directly relate to the system under test and that the current project release has no effect on them. Regression testing ensures that changes made to a module do not have an impact on the parent application’s integrated application’s overall functionality. To achieve successful regression testing, a specified collection of test cases should be accepted depending on their complexity, and a Test cases repository should be built. And the specific test should be updated anytime a new feature is added to the release.

Step 5: Performance Testing

The next step in mainframe testing is performance testing. The aim is to uncover bottlenecks in key areas like front-end data, upgrading online databases, and protecting the application’s scalability during performance testing. One may face the following performance problems in Mainframe applications-

• The internet response time may be slow, causing user dissatisfaction.
• Batch jobs and backend processes can take longer than expected, limiting online users’ access to the system.
• Issues with scalability.

To fix the issues listed above, run the application through the following tests-

• Parameters for system integration.
• Coding for application and database design.
• Parameters of the system and the database.
• Back-end job scheduling.

Step 6: Security Testing

Threats, hazards, and vulnerabilities are evaluated, and remedial actions for applications and networks are recommended. Use cases in identity and access management, risk and compliance management, data protection, and privacy policy adherence should all be included in the security testing. To put it another way, security testing is done to see how well an application is designed and constructed to withstand anti-security attacks. The two types of security systems that should be tested are mainframe security and network security. One must test the following factors during security testing-

• Authorization
• Integrity
• Authentication
• Confidentiality
• Availability

Step 7: Agile Methodologies

The Agile methodology is used to simplify the gradual development of applications and responds to modification quickly. 

Types of Mainframe Manual Testing

 Mainframe Manual Testing is divided into two parts:

• Online testing: The member enrollment screen is used for online testing. The database is checked using data input through the displays, just like a web page.
• Batch job testing: This procedure is tested in two phases- each job is validated independently, and the integration between the tasks is validated by supplying an input flat file to the first job and validating the database. (Because of the added caution, intermediate results must be confirmed.)

Mainframe Automation Testing Tools

The following are some of the mainframe automation testing tools:

1. Subject7

Subject7 is a simple, cloud-based SaaS test automation platform that delivers end-to-end test mechanization via a sequence of instructions. It’s a well-known name among the current automation solutions. This script-free, language-agnostic tool with an easy online interface was created to assist both non-technical testers and test automation experts in quickly creating complicated automated test scenarios.

Features:

• REST, JIRA, and Jenkins-competent DevOps pipelines are all simply integrated.
• Appium and Selenium are open-source standards that are used to test mobile and online applications.
• The web interface is simple to use, making it suitable for non-programmers.

2. LambdaTest

LambdaTest is a cross-browser testing platform that is scalable, cloud-based, and designed for both manual and automated software testing. It lets you test your public or locally hosted website or web app on more than 2000 different browsers, browser versions, operating systems, and resolutions. It provides a rapid preview of how the site will seem and allows one to test the layout on 36 various devices with just one click. On top of that, the platform lets one run Appium and Selenium scripts on a scalable online Selenium Grid across mobile browsers on both iOS and Android.

Features:

• Fully automated and interactive in real-time.
• The user interface is fantastic and simple to use.
• For real-time testing, a large number of browsers and mobile devices are available.

3. HeadSpin

Another popular Connected Intelligence Platform is HeadSpin, which provides mobile, 5G, web, and IoT applications. It unifies network, application, and device monitoring and testing by integrating with all automation and testing frameworks. One can test, monitor, and analyze any application running on any device, on any network, or anywhere in the world using HeadSpin.

Features:

• Profiling and debugging of local code.
• Debugging from afar.
• Testing for localization.
• There are almost 500 tests running in parallel.
• On a shared cloud device, there is access to more than 300 devices in more than 30 countries.

4. QTP

Quick Test Professional (QTP) now called UFT is used to test functional regression test cases of the web-based application. This is used to perform automated functional testing seamlessly without monitoring the system. 

• The tool is used for functional, regression, and service testing.
• It is used to test web, desktop applications, and client-server.
• There is a UFT extension in Chrome Store.
• Some of the newly supported technologies are JDK 1.8, and XenDesktop 7.
• UFT supports browsers Windows 8.1, Windows Server 2012, and Safari.

5. REXX

REXX is an interactive programming language that can execute system commands such as TSO, ISPF, etc. It is easy to use by experts and casual users.

• It has the capability to issue commands to its host environment.
• It has the capability to call programs and functions written in other languages.
• It has convenient built-in functions.
• It has the debugging capability. 

Best Practices For Mainframe Testing

1. Dry run of job: Doing a dry run of the job under test is always a smart idea. Empty input files are used for the dry run. This procedure should be undertaken for any jobs that are affected by the test cycle changes.
2. Complete test task setup: The test task setup should be completed well in advance of the start of the test cycle. This will aid in the early detection of any JCL errors, saving time during execution.
3. Set auto-commit to NO: Always set auto-commit to “NO” when accessing DB2 tables through SPUFI (the emulator’s option for accessing DB2 tables) to avoid unintentional updates.
4. Confirm technical inventory: Don’t underestimate the importance of effective project management and solution architect support for your project. Typically, these projects focus on applications that have been critical to the business for a long time. Confirming technical inventory and obtaining test and use case data are the two greatest time and expense drivers for mainframe migration initiatives. Make sure that your expertise is available and invested in the project.
5. Create required data in advance: It is considered best practice to create test data in advance of the test cycle and should be checked for completeness of the data.

Mainframe Testing Challenges and Troubleshooting

Every type of testing is a succession of trials and errors until you find the best system possible. Testing on mainframes is no different. Throughout the process, the testing team will be confronted with problems or troubleshooting. Some concerns that have been often reported by testers are discussed below, as well as a suggested approach that might be used to discover a solution.

1. Mismatch requirements and handbook

Although a user handbook or training guide may be available, these are not the same as the stated requirements.

Solution: The testing team should be involved in the Software Development Life Cycle from the moment the system’s requirements are defined. They will be able to verify that the criteria being specified are testable and feasible if they are involved early in the process. This saves the team time, money, and effort while also ensuring that the Software Development Process does not stall during the testing phase.

2. Identifying required data

There may be times when current data should be utilized to meet a specific need. Identifying the required data from the available data can be difficult at times.

Solution: Homegrown tools can be used to set up data as needed. Queries should be developed ahead of time to retrieve existing data. In the event of a problem, a request for the creation or cloning of required data can be made to the data management team.

3. No impact analysis

It’s possible that the code impact will completely alter the system’s appearance and functionality. Changes to test cases, scripts, and data may be necessary.

Solution: Impact analysis and a scope change management strategy should be in place.

4. Ad-hoc Request

It’s possible that faults with upstream or downstream applications will demand end-to-end testing. These unanticipated demands have the ability to derail the testing process’s pre-determined timetable by adding time, effort, and other resources to the execution cycle.

Solution: To prepare for unforeseeable issues throughout the testing process, automation scripts, regression scripts, skeleton scripts, and any other backup plans should be ready to use as soon as a problem arises. This cuts down on the total amount of time and work required to complete the project.

Benefits of Mainframe Testing

The following are some of the benefits of successfully completing the mainframe testing:

1. Optimized resource usage: It makes the most of the resources available and utilizes resources optimally.
2. Avoid duplicate rework: It assists in avoiding duplicate rework.
3. Improved user experience: It improves the overall user experience.
4. Reduced production time: It cuts down on production downtime.
5. Increased customer retention: It assists us in increasing customer retention.
6. Reduced IT operations cost: It also assists us in lowering the overall cost of IT operations.