Previous Process
Masters graduates in Business Analytics are trained to use Jupyter Notebooks to clean data through the 'Explorative Data Analysis' (EDA) method. 90% of time and effort is speant on this cleaning stage. Once it's done the dataset is clean.
A Not on Messiness
School will teach us there's one correct answer. They will give us a clean problem and expect a clean solution. Real datasets are not found pre-fabricated. They are grown, they need to be pruned, updated with new species added. As an example, the Menacon dataset is so wild and messy, we are constantly looking to improve on translations, add more fields, add more derived fields (intelligence) - we can only get closer to the signal.
Agentic EDA
Codex + Jupyter Notebook 'agentic skill'. For small datasets. For larger datasets we're using VS Code remote on an Intel Nuc stuffed with RAM sticks. There is no one app, one process solution.
New Working Process
Codex can handle EDA, implementing a number of ML algorithms that you suggest directly on the data. We suggest Jupyter Notbebooks are still used as they allow for human recreation of the results - which is still needed at this point for true understanding of intelligence. By true understanding, that creativity, or the direction that you want to take your project has to come from you, and it can only come from you if you truly know what you're have in front of yu.
Bottlenecks to Investigate Further
1. How do we further remove the human from the loop, which is now bottlenecked by the need to review the EDA? 2. How best to set up an agent which is trained in your particular preference of EDA and analytics in general. This includes favoured processes and their order, as well as the outputs - on-brand graphs and reporting. 3. How to quickly switch between different hardware, to keep folders tidy. 4. How to live, no revel in uncertainty.