This is the second blog in my series covering the SentryOne free Unit Test generator extension. In Part 1, I covered what the Unit Test Generator does, and in this post, I’ll talk about why we chose to implement it.
Before we dive back in, we invite you to download the extension and give us any feedback you may have, as it’s an open source tool, and we welcome all contributors!
Automated testing is incredibly important; the developer community understands this. At SentryOne, we focus our efforts on improving the lives of data professionals, and hence one of our products, SentryOne Test, is focused on enabling data testing without the heartache. But how do we go about testing the products we make?
Data testing is an essential part of what we do. But unit testing is also incredibly important as software grows, so we aim for a high level of code coverage from our unit tests. We also firmly believe in testing during development. It is harder to test code that was written a month ago than it is to write the tests as we go. Testing during the development process also helps us write testable code.
With that comes a cyclic rhythm of develop, validate, develop, validate—which, as you start to test smaller units of code, blur into one. If you get into the habit of testing each class and method as they grow, it’s less of a cycle and more of a single process. We want to keep up the pace of that rhythm, maximising productivity, and solving engineering challenges to provide products that fulfill the needs of our customers. That’s the need from which the SentryOne Unit Test Generator was born.
The general aims were to:
- Reduce engineering time spent scaffolding unit test code
- Test as close to development as possible
- Minimise having to manually write the boilerplate code that testing requires
- Cover the simple cases, which should become automatic and keep the developer in the mindset of the code that they are writing
- Emit boilerplate code for testing the complex cases
A Little History
We already had a Unit Test Generator as a Visual Studio extension. It was based on the NUnitGenAddin that originated from Novell, although it was heavily customised for our own needs. Its development could be politely described as organic—an hour here or there spent on it provided very useful functionality. However, it had significant drawbacks as time went on:
- It didn’t have AsyncPackage support. Visual Studio, from 2019.1 onwards, will only load packages that derive from AsyncPackage, which allows some of the package initialization to be done in the background, but that’s a topic for another day.
- The old extension used the EnvDTE CodeClass2 model to query information about the types for which to generate tests. This extension has been deprecated for a while and doesn’t understand a lot of the newer language features.
- To generate code, we used the standard .NET CodeDom model—and again, this hasn’t kept up with C#’s development. Arguably, it never supported a full set of language features (the inability to emit a while loop, for example). One of the benefits of CodeDom is the ability to emit code for C# or VB.NET through a single model—but we didn’t need to support VB.NET.
- As functionality grew with many time-limited enhancements, the code became a mess. Adding new functionality was a fight both against the existing code structure and the technology on which it relied.
As the extension aged, the desire to do something about those limitations became stronger, and it was evident that simply finding an hour or two here or there wouldn’t cut it.
Innovation Sprint
Cue the SentryOne Engineering Innovation Sprint. This was a sprint that we dedicated to the generation of new ideas—whether they took the form of new features, new products, or new processes. One of my co-workers, Alex Yumashev, suggested that we spend some time extending the Unit Test Generator to be able to add tests for methods added to classes that already have generated tests.
This initiative would not only support us in re-visiting code and refactoring but would also enable generating test classes earlier in the process. Then as you add method prototypes, tests for those methods can be generated even before the method bodies are implemented—which transforms the extension into something that also supports a test-driven development (TDD) flow. I thought that was a great idea, especially seeing as the opportunity afforded a larger block of time to work on it than we previously had.
Part of the preparation for the Innovation Sprint was a triage process for evaluating the validity of ideas. We encourage SentryOne employees to come up with ideas that positively impact our business, but we needed some checks and balances on whether this particular effort was likely to yield a positive result. As a business, we needed to ensure the Innovation Sprint was likely to finish with real value delivered.
One of the questions asked was about using an existing extension already available and extending that instead. It was a fair question—and these are the reasons that we chose not to take that route:
- While there were some great extensions out there, they generally fell short in a couple of areas:
- The tests generated by our old extension were already more valuable than the output of most generators. For example, automatic null checking of constructor parameters and checking that those properties are correctly initialised with constructor values.
- We didn’t find any that support augmenting existing test classes—and not having that functionality would have missed the point of the Innovation Sprint idea in the first place.
- We were willing to trade flexibility for greater functionality. While some extensions allow you to modify templates so that you can completely customize the output, that function reduces how much you can add to the output in terms of automatically testing the basic cases. Remember that one of the reasons we wrote the original extension was to keep the developer’s mindset on the functionality they were implementing, rather than typing.
- We also wanted to be a bit more ambitious. Aside from the ability to generate test cases that cover basic scenarios, we wanted to be able to cover the cases that are usually harder. For example, when attempting to generate tests for an abstract class, all the generators we found simply said they couldn’t. As our Vice President of Human Resources, Jenn Miller, often says, one of our company mantras is “Challenge Accepted.” So, in this case, we generated a test class that was derived from the class under test and worked out which members remained abstract, such that we needed to emit default bodies for them.
One thing that we did want to replicate from other projects, however, was the support for multiple frameworks. SentryOne acquired Pragmatic Works Software in 2018, and so there were two different code bases with different test frameworks. While standardisation is great, spending large quantities of time refactoring old code bases to use a different test framework isn’t something that would deliver value to our customers. The support for multiple frameworks suddenly became a much bigger factor, as it allows the extension to be used across the whole business.
Starting a Revolution
As I said previously, the primary goal of the SentryOne Unit Test Generator started as, “We would like the unit test generator to be able to add new test methods for existing classes.” When we ran the Innovation Sprint, engineers could sign up for the item they were most interested in. We had five people sign up. With that many hands on deck we now had the opportunity to re-write the extension completely, and a plan to address much of the desired functionality, including:
- We could form the basis of the extension around AsyncPackage, and make sure that everything adhered to current best practices from a Visual Studio extensibility perspective.
- We could avoid the pitfalls of the legacy technology that the old extension was based on and move to using Roslyn for model inspection.
- In summary, we could complete the sprint with functionality that had value on day one, in a healthier code base, lacking the technical debt of the old extension.
Stay Tuned for Part 3
I hope you enjoyed this second post on the open source SentryOne Unit Test Generator, which covered our motivations behind the creation of the tool. The “getting started part” of any project can be difficult, and it’s especially true for creative, technical projects. Finding the balance of scope and feature set, as you aim to deliver value is never easy. The SentryOne Innovation Sprint ended up being the perfect opportunity to tackle this project!
In the upcoming third and final post in my series on the SentryOne Unit Test Generator, I will be diving more into the technical aspects of how we built the extension. In the meantime, we invite you to download the extension and contribute any ideas you have on improving the tool! Thanks for reading.
