Briefly, ReproModel helps the AI research community reproduce and compare AI models faster.

The current prototype helps them to modularize their development and compare the performance of each step in the pipeline in a reproducible way.

The next version will help researchers to build upon state-of-the-art research faster by just loading the previously published study ID. All code, experiments, and results would already be verified, and stored in the system.

I'm an AI scientist, and while writing my PhD, I realised I'm spending most of my time trying to verify models and run codes from existing papers to evaluate it, instead of actually building my models and optimising them. While there are papers that include code snippets, creating a comparable version to the proposed approach is often not feasible due to differences in parameters, data loaders, or optimizers, for example.

To address that issue, I've put together multiple internal tools to speed up processes, and realised how valuable this can be for research on a larger scale. I've since structured my efforts and partnered up with colleagues, and we spent our free time honing these tools. I’ve received stellar feedback so far from the fellow researchers I presented our toolbox to.

To compare the results of my models to already existing ones, I’d have to copy their code, create a mini version of their script, copy all their parameters, then adapt my data and preprocessing to their exact values. This tedious workload is the norm, and it takes days and weeks. The caveat is, even with all this effort, you’re looking at a coin-flip, where the results may end up unusable due to differences in the models’ parameters. What makes what we built unique is that we are streamlining this process to be applied in a much more detailed way for every paper. That way, models and results are available to all researchers and they can quickly adapt from previous research. By automatically identifying and adapting the structure of the code into a defined template, everything is standardised and can be reproduced and compared. This effort is too redundant and time-consuming for any single researcher to perform on each individual research project.

The toolbox is in the frontend a no-code solution that works simply through multiple input boxes and dropdown menus. These menus allow users to select their parameters in the AI development process.

We've recorded this quick 2-minute walkthrough of every step of the process. Check it out here

The next iteration of the toolbox is expected to introduce intra-training possibilities, using cloud services. This allows users to rely on the toolbox to fully train models from scratch, extract results, and benchmark them in no time.

We're also working on creating an experiment description generator through a custom LLM embedded in the toolbox. This tool produces descriptions not just about the experiment itself, but also the results and findings, allowing scientists to focus on the development even further.

We will add more features in the future from the most upvoted feature-requests on our GitHub page.

We came up with this toolbox as a result of our personal frustration with the lack of reproducibility on the space. We've been using it consistently for the past few months, and realised the value it adds to our efficiency. The time-saving potential is huge for every researcher, and it opens up the space to focus on the more important aspects, i.e doing research and optimising models.

We hope fellow scientists find the same value we found, and help us improve our toolbox for the benefit of all the scientific community.