MIT Researchers Develop an Efficient Way to Train more Reliable AI Agents

Fields ranging from robotics to medicine to political science are attempting to train AI systems to make meaningful decisions of all kinds. For instance, utilizing an AI system to intelligently manage traffic in an overloaded city could help vehicle drivers reach their destinations faster, while improving security or sustainability.

Unfortunately, teaching an AI system to make excellent decisions is no easy job.

Reinforcement learning designs, which underlie these AI decision-making systems, still often stop working when faced with even little variations in the tasks they are trained to carry out. When it comes to traffic, a design might have a hard time to control a set of crossways with various speed limits, varieties of lanes, or traffic patterns.

To boost the reliability of reinforcement knowing models for complex tasks with variability, MIT researchers have actually introduced a more effective algorithm for training them.

The algorithm tactically selects the finest jobs for training an AI agent so it can efficiently perform all jobs in a collection of associated tasks. When it comes to control, each job could be one crossway in a task space that includes all intersections in the city.

By focusing on a smaller sized number of intersections that contribute the most to the algorithm’s general effectiveness, this method maximizes performance while keeping the training expense low.

The scientists found that their strategy was in between five and 50 times more efficient than standard techniques on an array of simulated tasks. This gain in performance assists the algorithm learn a much better option in a faster way, eventually improving the efficiency of the AI representative.

“We had the ability to see unbelievable performance enhancements, with a really basic algorithm, by believing outside package. An algorithm that is not extremely complicated stands a better possibility of being adopted by the neighborhood because it is simpler to implement and much easier for others to understand,” says senior author Cathy Wu, the Thomas D. and Virginia W. Cabot Career Development Associate Professor in Civil and Environmental Engineering (CEE) and the Institute for Data, Systems, and Society (IDSS), and a member of the Laboratory for Information and Decision Systems (LIDS).

She is joined on the paper by lead author Jung-Hoon Cho, a CEE college student; Vindula Jayawardana, a college student in the Department of Electrical Engineering and Computer Technology (EECS); and Sirui Li, an IDSS college student. The research study will be provided at the Conference on Neural Information Processing Systems.

Finding a happy medium

To train an algorithm to control traffic signal at lots of intersections in a city, an engineer would usually pick between two main techniques. She can train one algorithm for each crossway individually, utilizing only that intersection’s information, or train a larger algorithm using information from all crossways and after that apply it to each one.

But each technique includes its share of disadvantages. Training a different algorithm for each job (such as a provided crossway) is a lengthy process that needs a massive amount of data and computation, while training one algorithm for all tasks typically leads to subpar performance.

Wu and her collaborators looked for a sweet area between these 2 approaches.

For their technique, they pick a subset of jobs and train one algorithm for each job separately. Importantly, they tactically choose specific tasks which are probably to improve the algorithm’s overall efficiency on all jobs.

They leverage a typical technique from the reinforcement knowing field called zero-shot transfer knowing, in which a currently trained model is used to a new job without being further trained. With transfer knowing, the design often performs incredibly well on the brand-new neighbor task.

“We know it would be perfect to train on all the tasks, however we wondered if we could get away with training on a subset of those tasks, apply the result to all the tasks, and still see a performance increase,” Wu states.

To determine which tasks they need to select to optimize expected efficiency, the scientists established an algorithm called Model-Based Transfer Learning (MBTL).

The MBTL algorithm has two pieces. For one, it models how well each algorithm would perform if it were trained independently on one job. Then it designs just how much each algorithm’s performance would break down if it were transferred to each other task, a concept referred to as generalization efficiency.

Explicitly modeling generalization performance permits MBTL to approximate the value of training on a new task.

MBTL does this sequentially, choosing the task which leads to the highest efficiency gain first, then selecting additional jobs that offer the greatest subsequent minimal enhancements to total performance.

Since MBTL just concentrates on the most appealing jobs, it can significantly improve the efficiency of the training process.

Reducing training costs

When the scientists tested this method on simulated jobs, including controlling traffic signals, managing real-time speed advisories, and executing numerous classic control tasks, it was five to 50 times more efficient than other methods.

This indicates they might come to the exact same service by training on far less information. For example, with a 50x efficiency increase, the MBTL algorithm could train on simply 2 jobs and achieve the same performance as a standard approach which uses data from 100 tasks.

“From the point of view of the two primary techniques, that means information from the other 98 tasks was not needed or that training on all 100 jobs is puzzling to the algorithm, so the efficiency ends up even worse than ours,” Wu says.

With MBTL, including even a percentage of additional training time might lead to much better performance.

In the future, the scientists plan to develop MBTL algorithms that can reach more intricate problems, such as high-dimensional task areas. They are also interested in applying their method to real-world issues, specifically in next-generation mobility systems.