UNIVERSITY PARK, Pa. — A novel algorithm to solve big data resource sharing problems over large networks, developed by researchers in the Penn State College of Engineering, may also have implications for energy savings and data security.
The recent work led by Necdet Serhat Aybat, associate professor of industrial engineering, was published in the Society for Industrial and Applied Mathematics Journal on Optimization.
“Sometimes, when you minimize cost in one part of a network that has common resource constraints, it may skyrocket the cost in another part,” Aybat said. “Through this algorithm, we found a new way to efficiently minimize cost across the whole system in a decentralized manner.”
Modern society’s wealth of big data creates such high levels of information that are often difficult to process quickly and safely, because they require significant energy and bandwidth.
The traditional method of centralized optimization — gathering all of the data into one place for analysis — can be resource-expensive for large datasets because of the required memory storage and processing power. This traditional way of computation also raises concern for potential privacy issues. If the centralized system breaks, all of the data is at risk for exploitation.
To improve the process of analyzing big data, Aybat’s algorithm efficiently computes optimal resource sharing over a decentralized system that interacts over a communication network. Rather than compiling all of the data in one location, the system breaks out the information into various agents, or independent computing modules. Each agent is responsible for solving one task that affects the whole system.
To heighten the privacy of the information in the system, the agents are only aware of their own task and their teammates’ messages, meaning that they are unaware of a neighbor’s task. Once an agent solves its job, the agent only passes on the answer to its neighbors. This process repeats itself until every agent agrees on a common optimal resource allocation decision.
Decentralized optimization over communication networks has garnered attention for its use in a range of areas such as coordination and control in drones, bandwidth estimation in wireless sensor networks, machine learning data analysis and power control in cellular networks.
“The complication arises when neighbors are changing,” Aybat said. “If agents are moving, it could possibly cause the communications network to change over time. Since the agents can talk to only certain [other] agents at any given time, and you want to minimize the system cost, it becomes a difficult problem. You’re trying to carefully steer the information exchange among agents to share the scarce common resources while collectively minimizing the total system cost.”
The proposed algorithm, utilizing the decentralized optimization method involving multiple rounds of communication at each repetition, enables these agents to appropriately divvy out the common resource among them in such a way that accomplishes the goal.
“Working on this project has shown me the great applicability of both the methodology of mathematical techniques and its application in the real world,” said Erfan Yazdandoost Hamedani, a doctoral student in Aybat’s lab and co-lead author of the paper. “This work can be useful in the area of machine learning because this domain generates large amounts of potentially sensitive data.”
The National Science Foundation’s Directorate for Engineering within the Division of Civil, Mechanical and Manufacturing Innovation, and the Army Research Office supported this work.