- February 16, 2024
- Research
Scientists have invented an innovative algorithm-based approach that allows digital systems to predict trends in data and improve the storage process. Their findings suggest that such prediction can speed up data processing by as much as 40% when applied to real-world data collected.
A paper detailing their findings has been uploaded to the arXiv digital preprints platform and was presented at the Neural Information Processing Systems Conference (NeurIPS) in December 2023. The team, which includes experts from Carnegie Mellon University and Williams College, said the breakthrough could pave the way for fast databases and simplified data storage.
In their research, they focused on a typical data structure known as a list notation array, which sequentially organizes information in a device’s memory. This method of sorting data simplifies the search process, just as arranging names alphabetically in a directory facilitates quick identification. However, maintaining such an orderly sequence at all times, especially when incorporating new data, is a challenge. Previously, digital structures had to be planned in the context of the worst-case scenario, often changing information to accommodate incoming records, which took a lot of time and resources.
Using this new artificial intelligence technique endows these data structures with predictive power. The system scrutinizes recent data trends to predict what might happen next. “This approach gives digital infrastructure the ability to predict and dynamically improve its performance,” said Aydin Niaparasat, study co-author and doctoral student at Carnegie Mellon’s Tepper School of Business. “Our findings suggest a simple trade-off: the more accurate the predictions, the more effective the function. And this is true even when the forecasts are significantly inaccurate.”
The scientists made the tool available with the publication of their study, providing access to their software code for community use. They believe that this progress could herald the broader application of intelligent machine learning in the field of computational architecture. Elements such as search trees, hash maps, and network graphs can potentially increase their functionality by predicting trends in input data. The research team expects their work to inspire inventive designs for algorithms and data processing systems.
Benjamin Moseley, researcher and adjunct professor at the Tepper School, added, “Improved prediction can mean faster databases, better data center performance, and smarter system operation. We have proven that prediction can outperform traditional design constraints. However, this is only the tip of the iceberg – there are still many opportunities to be explored in this field.”
