R&D in an Agile workflow

The Agile Manifesto says that the ‘better’ way of developing software is to focus on working software over comprehensive documentation. A lot of people think Agile means no documentation required. However, if you’re claiming R&D through the R&D Tax Incentive then there is a legislated requirement to keep contemporaneous documents of experimental activities and associated costs.

The agile approach shouldn’t mean keeping zero documentation, instead focus on keeping your documentation lean and focused. So what documentation do you need to keep in order to make an R&D claim? Legislation says you need to keep up-to-date records which identify how and why experimental activities are being conducted. We’ve summarised the information you need to record and how regular documentation might fit into typical agile processes.

Zoom out for the big picture

Agile approaches encourage massive problems to be broken down in to tiny bite size pieces often known as user stories. This encourages regular code commits and a sense of velocity in development. However, this approach can make it harder to tell an R&D story that captures the bigger picture of a year’s worth of development.

R&D is all about “generating new knowledge”, which is not common lingo in software development circles. Phrases you do hear are ‘technical uncertainty’ or ‘technical risk’. What is a technical uncertainty? Well, this usually means developers have surveyed the existing solutions and there isn’t one that directly translates to the one their now trying to solve. In other words, there is an observed knowledge gap and the planned development will generate new knowledge.

These technical uncertainties are often discussed regularly and in great detail within the team but never written down because there isn’t an agile ceremony which demands it. User stories describe a set of problems from the perspective of the user but don’t prescribe the solution for it. Internal documentation usually captures only the working solution.

A good approach to is to use the time set aside for planning and retrospectives to keep an evolving document of technical uncertainties. Planning might throw up an array of new uncertainties, particularly planning in the early stages of a project, but every planning session may turn up something useful. Keep a list and come back to them. In your retrospectives, ask the questions:

• Have these uncertainties been resolved?
• How will you test the uncertainty?
• Did any of the work in this sprint get you closer?
• With new understanding, can any be expanded on or grouped together?

Keeping an ongoing and evolving list of uncertainties, and the things you did in a sprint’s worth of development to solve them, will make future R&D application easier to write. It helps structure a broader picture of development than atomised user stories and provides clear evidence that the reason you conducted this development for ‘the purpose of generating new knowledge’.

Data, metrics, results and frame-grabs

The most important area of documentation to keep throughout development is metrics that relate to your technical uncertainties. This will be different from project to project, but might include;

• CPU usage,
• Latency of a particular endpoint,
• Screenshots showing quality,
• Database usage

Results should be kept not just at the conclusion of work when there is a working solution, but for each prototype throughout development. This includes early attempts prior to putting a solution live, failed approaches which may never have been tested outside of local environments and intermediate prototypes which show modest improvements.

Keeping track of results shows a systematic progression of work. This is not just a legislative requirement, but also good development practice which will enable you to evolve your product on an ongoing basis. Articulating which metrics are important in solving a technical uncertainty will help keep development work focused and again fortifies your R&D claim with evidence.

For more information you can read the government’s software specific guidance.