In a Test Automation Framework, it’s very important to have a Summary Report of test execution. There are many report options such as Cucumber Report, Extent Report, etc. which are pretty detailed reports, but most of the time these reports are generated either inside the CI/CD tool (i.e. Jenkins, etc.) or within the local machine on which we are executing the tests. In such a case, either we need to mention the report link or attach the entire report folder in the summary report email. The problem with this is, most of the time, other stakeholders (such as Project Manager, Business Analyst, Program Manager, Business Leaders, etc.) don’t have time to login to Jenkins or open the report folder and check the report. We would definitely need a detailed HTML Summary Report that can be sent in the email body itself to all the stakeholders so that they can easily understand the execution status.
Behavior Driven Development (BDD) Test Automation Framework is very popular nowadays. Whoever is using BDD Test Automation Framework, it’s very likely that they use Cucumber to write feature files. Mostly the framework generates a cucumber-report.json file as a summary of the entire execution. In this article, we will look into how we can generate an HTML Summary Report using Java. We will also discuss the required dependencies to read JSON files and how can we send this report from Jenkins as an email body.
Let’s start with the required dependencies to read a JSON file, we have used json-simple here. Below is the dependency that you need to add in pom.xml if you are using a Maven Project.
XML
<dependency> <groupId>com.googlecode.json-simple</groupId> <artifactId>json-simple</artifactId> <version>1.1.1</version></dependency> |
Next, create a class ScenarioDetails, this is used to hold the status and failure message (if any) of a scenario. The structure of ScenarioDetails is given below.
Java
public class ScenarioDetails { boolean status; String faiureMessage; public ScenarioDetails(boolean status, String faiureMessage) { this.status = status; this.faiureMessage = faiureMessage; }} |
Please find the below entire code in Java to read the cucumber-report.json file and generate the Summary HTML Report. At the high level, the following are the steps performed in the code:
- Parse the cucumber-report.json file.
- Get the root JSONArray. Typically the size of the root JSONArray signifies the total number of feature files involved.
- Iterate each feature file.
- For each feature file, iterate over each scenario.
- For each scenario, retrieve the steps involved and validate the result of each step. Identify the scenario as pass or fail accordingly.
- Store the consolidated scenario details in List<HashMap<String, String>>.
- Use the above-consolidated details to generate the HTML report.
Java
import java.io.*;import java.util.*;import org.json.simple.JSONArray;import org.json.simple.JSONObject;import org.json.simple.parser.JSONParser;import org.json.simple.parser.ParseException; public class GenerateSummaryReport { public static List<HashMap<String, String> > allFailedScenarios = new ArrayList<>(); public static void main(String[] args) throws IOException, ParseException { List<HashMap<String, String> > consolidatedScenarioWiseResult = new ArrayList<>(); String scenarioName; String featureName; JSONParser parser = new JSONParser(); // Parse the cucumber-report.json file. Please // change the path according to your project Object obj = parser.parse(new FileReader( System.getProperty("user.dir") + "\\target\\cucumber-report\\cucumber-report.json")); // rootArray defines the number of feature files in // the cucumber-report.json JSONArray rootArray = (JSONArray)obj; // loop through each feature file for (int featureID = 0; featureID < rootArray.size(); featureID++) { JSONObject rootMap = (JSONObject)rootArray.get(featureID); featureName = rootMap.get("name").toString(); // elementsArray defines the number of scenario // for a feature file JSONArray elementsArray = (JSONArray)rootMap.get("elements"); // loop through each scenario of a feature file for (int i = 0; i < elementsArray.size(); i++) { JSONObject scenarioWiseObject = (JSONObject)elementsArray.get(i); scenarioName = scenarioWiseObject.get("name") .toString(); // Validate if any step got failed in case // @Before step present If any step got // failed, break the loop and store in a // List<HashMap<String, String>> if (scenarioWiseObject.get("before") != null) { ScenarioDetails result = getScenarioStatus( (JSONArray)scenarioWiseObject .get("before")); if (!result.status) { HashMap<String, String> temp = new HashMap<>(4); temp.put("FEATURE_NAME", featureName); temp.put("SCENARIO_NAME", scenarioName); temp.put("STATUS", "FAIL"); temp.put("ERROR_MESSAGE", result.faiureMessage); consolidatedScenarioWiseResult.add( temp); continue; } } // Validate if any step got failed in // scenario steps If any step got failed, // break the loop and store in a // List<HashMap<String, String>> if (scenarioWiseObject.get("steps") != null) { ScenarioDetails result = getScenarioStatus( (JSONArray)scenarioWiseObject .get("steps")); if (!result.status) { HashMap<String, String> temp = new HashMap<>(4); temp.put("FEATURE_NAME", featureName); temp.put("SCENARIO_NAME", scenarioName); temp.put("STATUS", "FAIL"); temp.put("ERROR_MESSAGE", result.faiureMessage); consolidatedScenarioWiseResult.add( temp); continue; } } // Validate if any step got failed in case // @After steps present If any step got // failed, break the loop and store in a // List<HashMap<String, String>> if (scenarioWiseObject.get("after") != null) { ScenarioDetails result = getScenarioStatus( (JSONArray)scenarioWiseObject .get("after")); if (!result.status) { HashMap<String, String> temp = new HashMap<>(4); temp.put("FEATURE_NAME", featureName); temp.put("SCENARIO_NAME", scenarioName); temp.put("STATUS", "FAIL"); temp.put("ERROR_MESSAGE", result.faiureMessage); consolidatedScenarioWiseResult.add( temp); continue; } } // At this point the scenario would have // definitely passed, hence storing the // result as Pass HashMap<String, String> temp = new HashMap<>(4); temp.put("FEATURE_NAME", featureName); temp.put("SCENARIO_NAME", scenarioName); temp.put("STATUS", "PASS"); temp.put("ERROR_MESSAGE", "NA"); consolidatedScenarioWiseResult.add(temp); } } // Call the summary details method to get pass, // fail, pass percentage, fail percentage for each // feature file as well as in total List<HashMap<String, String> > summary = getSummaryDetails( consolidatedScenarioWiseResult); // below method generates the HTML report using the // summary data. generateHTMLReport(summary); } public static ScenarioDetails getScenarioStatus(JSONArray stepName) { try { JSONObject stepObject; JSONObject resultObject; // iterate over each steps of a scenario for (int i = 0; i < stepName.size(); i++) { stepObject = (JSONObject)stepName.get(i); resultObject = (JSONObject)stepObject.get("result"); // check for the status of the steps and if // any of the step is fail get the error // message if (resultObject.get("status") != null && !resultObject.get("status") .toString() .trim() .contains("passed")) { if (resultObject.get("error_message") != null) { return new ScenarioDetails( false, resultObject .get("error_message") .toString()); } else { return new ScenarioDetails( false, "Error Message Not Found"); } } } return new ScenarioDetails(true, ""); } catch (Exception e) { System.out.println( "Exception occurred while getting scenario status"); return new ScenarioDetails(false, "Exception - " + e); } } // Below method will accept the // consolidatedScenarioWiseResult and calculate the // summary to be put in HTML report public static List<HashMap<String, String> > getSummaryDetails(List<HashMap<String, String> > consolidatedScenarioWiseResult) { try { int totalScenario = 0; int totalPass = 0; int totalFail = 0; // Get all the feature file name first in a // HashSet to filter duplicate Set<String> allFeatureName = new HashSet<>( consolidatedScenarioWiseResult.size()); List<HashMap<String, String> > summary = new ArrayList<>(); for (int i = 0; i < consolidatedScenarioWiseResult.size(); i++) { allFeatureName.add( consolidatedScenarioWiseResult.get(i) .get("FEATURE_NAME")); } Iterator<String> featureIT = allFeatureName.iterator(); while (featureIT.hasNext()) { int total = 0; int pass = 0; int fail = 0; String featureName = featureIT.next(); for (int i = 0; i < consolidatedScenarioWiseResult .size(); i++) { if (consolidatedScenarioWiseResult .get(i) .get("FEATURE_NAME") .equalsIgnoreCase( featureName)) { total++; if (consolidatedScenarioWiseResult .get(i) .get("STATUS") .equalsIgnoreCase("PASS")) { pass++; } else { fail++; allFailedScenarios.add( consolidatedScenarioWiseResult .get(i)); } } } HashMap<String, String> temp = new HashMap<>(); temp.put("FEATURE_NAME", featureName); temp.put("TOTAL", "" + total); temp.put("PASS", "" + pass); temp.put("FAIL", "" + fail); double pass_per = ((double)pass / (double)total) * 100; double fail_per = ((double)fail / (double)total) * 100; temp.put("P_PER", "" + pass_per); temp.put("F_PER", "" + fail_per); summary.add(temp); totalScenario = totalScenario + total; totalPass = totalPass + pass; totalFail = totalFail + fail; } HashMap<String, String> temp = new HashMap<>(); temp.put("FEATURE_NAME", "TOTAL"); temp.put("TOTAL", "" + totalScenario); temp.put("PASS", "" + totalPass); temp.put("FAIL", "" + totalFail); double pass_per = ((double)totalPass / (double)totalScenario) * 100; double fail_per = ((double)totalFail / (double)totalScenario) * 100; temp.put("P_PER", "" + String.format("%.2f", pass_per)); temp.put("F_PER", "" + String.format("%.2f", fail_per)); summary.add(temp); return summary; } catch (Exception e) { System.out.println( "Exception occurred while getting summary : " + e); return new ArrayList<>(); } } // Below code will concat the required HTML Code with // all the summary details and generate the HTML report. public static void generateHTMLReport( List<HashMap<String, String> > summary) { String htmlCode = "<html>\n" + "<head>\n" + "\t<style>\n" + "\ttable,\n" + "\tth,\n" + "\ttd {\n" + "\t\tborder: 1px solid black;\n" + "\t\tborder-collapse: collapse;\n" + "\t}\n" + "\t\n" + "\t\n" + "\tth {\n" + "\t\ttext-align: center;\n" + "\t}\n" + "\t</style>\n" + "</head>\n" + "<body>\n" + "\t<table style=\"width:100%\">\n" + "\t <caption style=\"background-color:#000000;color:white\">Sample Project Name</caption>\n" + "\t\t<caption style=\"background-color:#4287f5;color:white\">Test Automation Summary Report</caption>\n" + "\t\t<tr>\n" + "\t\t\t<th>Feature Name</th>\n" + "\t\t\t<th>Total Scenario</th>\n" + "\t\t\t<th>Pass</th>\n" + "\t\t\t<th>Fail</th>\n" + "\t\t\t<th>Pass Percentage</th>\n" + "\t\t\t<th>Fail Percentage</th>\n" + "\t\t\t\n" + "\t\t</tr>"; for (int i = 0; i < summary.size(); i++) { htmlCode = htmlCode + "<tr><td>" + summary.get(i).get("FEATURE_NAME") + "</td>\n" + "\t\t\t<td>" + summary.get(i).get("TOTAL") + "</td>\n" + "\t\t\t<td>" + summary.get(i).get("PASS") + "</td>\n" + "\t\t\t<td>" + summary.get(i).get("FAIL") + "</td>\n" + "\t\t\t<td>" + summary.get(i).get("P_PER") + "</td>\n" + "\t\t\t<td>" + summary.get(i).get("F_PER") + "</td></tr>"; } htmlCode = htmlCode + "</table>\n" + "\t<br></br>"; if (!allFailedScenarios.isEmpty()) { htmlCode = htmlCode + "<table style=\"width:100%\">\n" + "\t\t<caption style=\"background-color:#4287f5;color:white\">Failed Scenarios</caption>\n" + "\t\t<tr>\n" + "\t\t\t<th>Feature Name</th>\n" + "\t\t\t<th>Scenario Name</th>\n" + "\t\t\t<th>Status</th>\n" + "\t\t\t<th>Failure Reason</th>\n" + "\t\t\t\n" + "\t\t</tr>"; for (int i = 0; i < allFailedScenarios.size(); i++) { htmlCode = htmlCode + "<tr>\n" + "\t\t\t<td>" + allFailedScenarios.get(i).get( "FEATURE_NAME") + "</td>\n" + "\t\t\t<td>" + allFailedScenarios.get(i).get( "SCENARIO_NAME") + "</td>\n" + "\t\t\t<td>Fail</td>\n" + "\t\t\t<td>" + allFailedScenarios.get(i).get( "ERROR_MESSAGE") + "</td>\n" + "\t\t</tr>"; } htmlCode = htmlCode + "</table>\n" + "</body>\n" + "\n" + "</html>"; } try { Writer fileWriter = new FileWriter( new File(System.getProperty("user.dir") + "/target/emailReport.html")); fileWriter.write(htmlCode); fileWriter.close(); System.out.println( "Report generated successfully"); } catch (Exception e) { System.out.println( "Exception occurred while generating the html code : " + e); } }} |
Here is a sample summary report that would be generated. We have deliberately failed one scenario to show how does the report looks like in case of failures. Please check the comments in the code to customize the report name, and change the cucumber.json report location, and the html report generation location.
Please note the above code is a java main function that can be invoked at the end of the execution using the Maven Exec plugin. Below is the code snippet that we need to add in pom.xml. Under configuration, we need to provide the class name (including the package name) along with classpathScope as a test.
XML
<plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>exec-maven-plugin</artifactId> <version>1.6.0</version> <configuration> <mainClass>com.report.GenerateReport</mainClass> <classpathScope>test</classpathScope> </configuration></plugin> |
Now, to execute the above at the end of the execution, simply add “exec:java” at the end of the maven command. Here is an example of a maven command.
clean verify -Denv=STOCK -Dtag=@STOCK_ANALYSIS -Dfeature=stock_Analysis -DforkCount=1 exec:java
Next, let’s see how to publish this report from Jenkins. To do that we can use Email Extension https://plugins.jenkins.io/email-ext/ plugin. There is a field Default Content in which you can provide the location of the html report as ${FILE,path=”target/emailReport.html”}.
