AI-Powered test writing for any codebase

@rawandshaswar|February 11, 2025 (5m ago)112 views

Test writing is a crucial part of software development. It ensures that your code works as expected and that it continues to work as you make changes. However, writing tests can be time-consuming and tedious, especially for large codebases. Any well written test suite should be comprehensive, covering all possible edge cases and ensuring that the code behaves as expected in all scenarios. This can be a daunting task, especially for complex codebases with many moving parts.

At FIB, we have well over 13,000 tests in our codebase to ensure every feature is well tested, covered and works as expected for the long run. Any new feature in our codebase requires a new set of test suites specific to it, and some of these can be repetitive, and some require you to think from the ground up.

Regardless, testing and QA are not something you can skip. While it is not user-centric, it is crucial to ensure that your application is reliable and works as expected.

The need for a tool like Cactus

In all of my development journey, I have found that tests are the most repetitive part of the process.

...but it doesn't have to be. I get it it's crucial to make sure your codebase is well-tested, but it doesn't have to be a time-consuming process.

with that being said, I have found that the learning curve for writing tests is minimal, and with that, and that the process of writing tests can be automated.

Introducing Cactus

What can Cactus do? It can create and automate the writing of tests using a zero logging LLM of your choice, with context provided by you. It can also generate tests for you based on your previous testing patterns.

• It supports drag and drop, and lives in your menu bar
• It saves you time on reduces errors that may arise when you write tests, manually.
• It follows common MVVM and MVVM-C patterns, and applies carefully crafted prompts to ensure that the generated tests are accurate and relevant to your codebase.

As mentioned, cactus works in your menu bar, when you open the app. Initially you will need to provide context for the LLMs to use and reference when writing your new tests. Say you want to follow a specific pattern employed by a use case. you’d simply select the whole directory for the use case from the tests, and the LLM will learn, and reference them when writing tests. under the hood this uses OpenAI and Anthropic LLMs that are reported to be decent at coding capabilites.

1. The story behind Cactus

Cactus was born out of a need to automate the writing of tests for our codebase. We have a large codebase with many moving parts, We wanted to find a way to automate this process and make it easier for our developers to write tests.

Ever since I joined FIB, I have spent countless hours writing tests, and getting them right is a challenging process, and because there are strict patterns one must follow. The learning gap in this process is minimal.

It is no secret that AI is really good at following instructions, and specifically writing code, and hallucinations is a sensitive topic, but with proper instructions, and previous patterns, writing tests would never be the same.

I created Cactus for myself, and decided to keep it for myself because at the time, It helped me deliver high quality tests that ensured edge cases were covered, and that the code behaved as expected.

After a few months of trial and run, I decided to share it with the team, while i cannot say it was a hit, it was a good tool to have in the arsenal.

2. Cactus is not magic

While it can be a really good tool to have at the side, given proper previous tests for the AI to learn from, it is not magic. It is not a replacement for manual testing, and it is not a replacement for QA. It is a tool that can help you write tests faster, and ensure that your code is well tested and reliable.

It is not a replacement for human judgement, due dillingence is still required, and it is important to review the tests generated by Cactus to ensure that they are accurate and relevant to the codebase.

3. In active use

Overall, during my tests I have found that Cactus can write tests really well, if given valid context.

Aside from the test writing capabilities, cactus have a WIP feature that learns from previous change requests given by the team during Code Reviews, and upon giving it a file, it can help you fix and address change requests, before they are potentially raised by the team.

4. Costs?

While the tool itself is internal (as of now), and is free to use, the LLM providers under the hood are not free, but they're not crazy expensive either

Cactus works with about 10-15 files as previous test context, and then given a new file, and upon generating a test suite, the costs may be under $0.05 or less (depending on the model)

5. Conclusion

Cactus is a powerful tool that can help you write tests faster and more accurately. It can save you time and reduce errors that may arise when you write tests manually. It is not a replacement for manual testing, but it can be a valuable tool to have in your arsenal.

Cactus was a small experiment that turned out really valuable to me, it is certainly the start of something bigger, and I am excited to see where it goes.