How to Successfully Manage AI Software Projects
The 4 Phases of AI Projects I Shared at VixulCon
tl;dr
I gave a talk at VixulCon, a conference for technology consulting companies, on an AI project methodology I’ve developed
Managing AI software projects is different from managing regular software projects because AI software is non-deterministic.
There are four main phases to AI projects: Discovery, Implementation, Enhancements, and Go-Live
I was in Austin, Texas, last month for a conference called VixulCon.
It was an extension of the technology consulting accelerator I did last Fall where I got to meet the founders and other cohort members in person.
Around 100 people showed up to this conference in Austin. It was a one-day affair with workshops, panel discussions, and speakers.
I presented on the lessons I’ve learned on how to run AI projects and enjoyed all the questions and engagement from attendees.
So today, I wanted to share a summary of the AI project methodology I shared at VixulCon.
The Four Main Phases of AI Projects
In the talk, I discussed how AI projects have four main phases:
Discovery (45%) - where we prepare all the data, prompts, and integrations needed to build the product out
Implementation (25%) - when we build out the AI software
Enhancements (25%) - when we improve accuracy and performance
Go-Live (5%) - when we deploy the tool and monitor the launch
Note that I have a phase 1.5 here because during implementation, I always ask clients which features they need for phase 1, and which ones can wait for phase 1.5.
It helps scope down the initial project while still giving clients the assurance that we will get to the other features eventually.
It also helps shorten the time to launch, which clients always want.
In terms of the time allocation split, the bulk of the time is spent during Discovery at 45%, Implementation at 25%, Enhancements at 25% and Go-Live at 5%.
This split is very different from regular software projects.
In general, they say that Discovery for regular software projects should take between 10-20% of the total time available.
So for example, 1 month on Discovery for a regular software project is justified for an Implementation lasting between 5-10 months.
However, this rule of estimation is inaccurate because AI Projects require a longer Discovery for one main reason:
AI Software is non-deterministic.
Why Regular Software is Different from AI Software
Regular software, given the same inputs, will have the same outputs.
But AI software is based on probabilities, so the outputs are not guaranteed.
Thus more time has to be spent during the “Enhancements” phase to improve accuracy, which is not necessary in regular software projects since accuracy there is 100%.
Furthermore, more experimentation is needed to test different prompts and technologies.
Here are some of the key milestones that often happen during Discovery.
Manage Expectations from the Very Beginning
The first key milestone happens before the project even begins, which is in the first meeting with the client to set expectations.
It’s critical to have an initial conversation to outline what the client can expect during the project, potential risks, and the iteration needed to improve accuracy because they may not understand what such a project entails.
A line I like to use is that “if ChatGPT can have accuracy and hallucination issues, than any software we build can also have the same issues as well.”
Benchmarking is a Critical First Step in AI Projects for 2 Reasons
Second, I would start Discovery with benchmarking.
Ask the client to give you a list of the types of inputs and outputs they expect the software to have.
This is important because it helps you more rigorously evaluate the performance of your software. If the software starts by answering 10/30 questions correctly but at the end answers 25/30 questions, you can say that the performance has improved.
Second, it also affects how you build the product.
Consider these two questions: “How many insurance providers are in California?” vs. “Write me an email to negotiate rates with X insurance provider.”
These two questions are actually very different and necessitate different system designs.
The first question is a counting question that relies on Text-to-SQL: Converting the question into a SQL query and executing it on a database to get the most accurate question.
The second question is regular language generation, so the two questions require different designs to answer.
During Discovery, I would also prepare the data necessary for augmentation or fine tuning.
Scraping can be very tedious and time-consuming, so I would generally recommend someone other than the implementation engineer to do this.
And finally, I recommend trying to reproduce the outputs in a test environment. For example, OpenAI has a playground environment where you can test the API’s.
If you can’t get the right answers during testing, you certainly won’t get it when you build it out.
Get Everything Working First Before Improving Accuracy
AI implementation is a fraction of the time compared to regular software building because if everything was prepared, coding up the project is typically not difficult.
I’ve seen the core backend of many of these AI projects consist of only a few hundred lines of code.
During Implementation, the goal is to get everything working end-to-end first, before moving to Enhancements to improve accuracy and performance.
The first reason is that it helps maintain project momentum.
Improving accuracy and performance can require a lot of iteration and be endless, since you can always continue improving performance.
So it’s better to save that to the end, timebox the performance improvements, and get an MVP working first.
Save Non-Critical Features for Phase 1.5
After we finish Enhancements, we are feature complete, and the heavy lifting is done.
Afterwards though, it’s important to account for some time to stick around and ensure the launch is successful. It’s not enough to just build it and hand it off, as clients often need some support at the end.
And Phase 1.5 is where we address everything out of scope for Phase 1.
How We Applied This to a Recent Project
In a recent project, a client approached me to build an AI-generated 24/7 radio show of celebrities speaking on modern day topics.
They had singled out one particular celebrity whose prolific library of past content gave us enough data to finetune an LLM on. Their popularity would also help this product spread among their existing audience.
The client was aiming to present this during investor meetings after.
We followed this methodology to a tee, where we spent:
3 Weeks on Discovery, where we scraped, cleansed, and prepared examples of the celebrity’s tone and unique ways of speaking to finetune the LLM with. We also tested different prompts to generate the right type of structure for the show
~2 weeks on Implementation (25%) - when we integrated the voice clones and built out a fine-tuned model for the client to interact with
~2 weeks on Enhancements (25%) - when we improved the accuracy, performance, and aesthetics of the tool
And 3 days with the Go-Live (5%) - when we deployed the tool and handed it off.
The client was so happy, they left us a 5-star review. You can read the full case study here.
Final Thoughts
The next time you work on an AI project, try dividing your project into the 4 distinct phases above.
Make sure to allocate the most time to Discovery, get everything working end-to-end before improving accuracy, and leave some time for the launch.
It will help you run your project more smoothly!
In the mean time, feel free to connect with me on LinkedIn or Twitter where I post daily content on AI and entrepeneurship.
You can also connect with me here on Instagram.
And, if you have any referrals for folks interested in building anything related to AI, let me know!
I offer a $1,000 referral bonus for every opportunity that closes (website link).
Thanks,
Michael