When George Heilmeier was the director of ARPA in the mid 1970s, he had a standard set of questions he expected every proposal for a new research program to answer. These have been called the Heilmeier Catechism. It’s a good exercise to answer these questions for an individual research project, too, both for yourself and as a way to convey to others what you hope to accomplish.
A nice set of questions to ask yourself when doing research. They are part of Heilmeier Catechism, who was the director of research at ARPA.
They’re also questions you can ask when you’re working on something that could be turned into a research paper (which is a lot of things in the computer world unsurprisingly).
Or when you want to execute a marketing or sales project that relies on a new technology.
Example of using Heilmeier’s Catechism
Below we will explore each question that is part of the Heilmeier Catechism in the context of a software development team TeamSaas working on a SaaS product adopting GraphQL for the API.
The questions to ask: Heilmeier Catechism in an example context
1. What is the problem? Why is it hard?
In our example, TeamSaas needs to cater to multiple clients and consumers of their API and maintain a maximum payload size of some amount. This is difficult because different clients/consumers have different needs (some may need a lot of data, others may need less data). TeamSaas is aiming to minimize the payload size and give clients/consumers of their API only the data they need.
2. How is it solved today?
Today, TeamSaas would use a REST API.
3. What is the new technical idea; why can we succeed now?
GraphQL is the new technical idea, it allows a client/consumer of an API to select and filter data before it is passed back. The reason it can succeed now is that GraphQL has an integration with the database and framework and language that TeamSaas is using; there are also client libraries that can consume a GraphQL API and they are (relatively) stable. The main reason though is that there is enough adoption and resources aimed at making GraphQL implementable and stable.
4. What is the impact if successful?
In our example, the impact for TeamSaas is that they will be able to send the exact data that each client/consumer of their API needs; additional impacts include a cleaner codebase on their server-side and on the client-side since the data is handled through a GraphQL layer and the API response doesn’t have to be transformed in an awkward/hacky way.
5. How will the program be organized?
TeamSaas could organize their project with a client developer and backend developer prototyping and developing one or two API calls using GraphQL and then further expanding from there.
6. How will intermediate results be generated?
In our example, TeamSaas can use their prototype to generate API responses and operate the new GraphQL APi in parallel with the old REST API.
7. How will you measure progress?
Progress for TeamSaas’s switch from a REST API to a GraphQL-based API can be measured by the number of clients/consumers using the new GraphQL API and by the number of complaints there are. If there are more clients switching and there are few complaints/feedback, then things are going smoothly. Additionally, the payload sizes for each request can be measured and averaged and compared in the 90th to 99th percentile range to the old REST API to ensure payload sizes overall are smaller. The total payload sizes delivered over a week or a month could be measured, aiming for a decreased payload size total in comparison to the REST API.
8. What will it cost?
This is an exercise for the reader; the cost of the project depends on the tasks and time estimates and the team.