【论文推荐】四川大学刘俊勇等：基于深度强化学习的城市配电网多级动态重构：一种云边协同架构

摘要

随着电力物联网的建设，城市配电网中智能设备之间的通信联络已经逐渐走向高速率、高兼容和低延迟的道路，这为城市配电网重构优化提供了可靠支撑。因此，本文提出了云边协同架构下基于强化学习的城市配电网多级动态重构方法以获得实时最优的多级动态重构方案。首先，文章讨论了多级动态重构方法的概念，包括馈线级、变压器级和变电站级重构。然后，将多智能体系统与云边协同架构相结合用以构建城市配电网多级动态重构的深度强化学习模型，其中云边协同架构可以有效支持多智能体系统进行“集中训练、分散执行”的运行模式并提高模型的学习效率。接着，对于多智能体系统，本文采用了离线学习与在线学习相结合的方法，以赋予模型实现策略自动优化和更新的能力。在离线学习阶段，本文提出了一种基于Q学习的多智能体保守Q学习算法以稳定学习结果并降低之后的在线学习阶段的风险。在在线学习阶段，本文提出了一种基于策略梯度的多智能体深度确定性策略梯度算法，以探索行动空间并更新经验池。最后，通过对真实世界445节点系统的仿真分析，验证了所提方法的有效性。

Deep reinforcement learning based multi-level dynamic reconfiguration for urban distribution network: a cloud-edge collaboration architecture

基于深度强化学习的城市配电网多级动态重构：一种云边协同架构

Siyuan Jiang¹, Hongjun Gao¹, Xiaohui Wang², Junyong Liu¹, Kunyu Zuo³

(1.College of Electrical Engineering, Sichuan University, Chengdu Sichuan 610065, P. R. China

2.China Electric Power Research Institute, Haidian District, Beijing 100192, P. R. China

3.Electrical and Computer Engineering Department, Stevens Institute of Technology, Hoboken, NJ07030, USA)

收听作者1分钟语音介绍

01 Deep reinforcement learning based multi-level

Abstract

With the construction of the power Internet of Things (IoT), communication between smart devices in urban distribution networks has been gradually moving towards high speed, high compatibility, and low latency, which provides reliable support for reconfiguration optimization in urban distribution networks. Thus, this study proposed a deep reinforcement learning based multi-level dynamic reconfiguration method for urban distribution networks in a cloud-edge collaboration architecture to obtain a real-time optimal multi-level dynamic reconfiguration solution. First, the multi-level dynamic reconfiguration method was discussed, which included feeder-, transformer-, and substation-levels. Subsequently, the multi-agent system was combined with the cloud-edge collaboration architecture to build a deep reinforcement learning model for multi-level dynamic reconfiguration in an urban distribution network. The cloud-edge collaboration architecture can effectively support the multi-agent system to conduct “centralized training and decentralized execution” operation modes and improve the learning efficiency of the model. Thereafter, for a multi-agent system, this study adopted a combination of offline and online learning to endow the model with the ability to realize automatic optimization and updation of the strategy. In the offline learning phase, a Q-learning-based multi-agent conservative Q-learning (MACQL) algorithm was proposed to stabilize the learning results and reduce the risk of the next online learning phase. In the online learning phase, a multi- agent deep deterministic policy gradient (MADDPG) algorithm based on policy gradients was proposed to explore the action space and update the experience pool. Finally, the effectiveness of the proposed method was verified through a simulation analysis of a real-world 445-node system.

Keywords

Cloud-edge collaboration architecture, Multi- agent deep reinforcement learning, Multi-level dynamic reconfiguration, Offline learning, Online learning.

Fig.1 Multi-level dynamic reconfiguration

Fig.2 Multi level dynamic reconfiguration system architecture of urban distribution network based on multi- agent deep reinforcement learning under cloud edge collaborative framework

Fig.3 Architecture of multi-agent deep reinforcement learning system

Fig.4 Algorithm comparison

Fig.5 Comparison of effects of different neural networks on performance

Fig.6 Comparison of model performance with or without offline reinforcement learning

本文引文信息

Jiang SY, Gao JH, Wang XH, et al (2023) Deep reinforcement learning based multi-level dynamic reconfiguration for urban distribution network: a cloud-edge collaboration architecture. Global Energy Interconnection, 1(1): 1-14

姜思远, 高红均, 王晓辉等 (2023) 基于深度强化学习的城市配电网多级动态重构：一种云边协同架构. 全球能源互联网（英文）, 1(1): 1-14

Biographies

Siyuan Jiang

Siyuan Jiang received his B.S. degree in electrical engineering from Nanjing University of Information Science and Technology, Nanjing, China in 2021, and he is currently pursuing the M.S. degree in Sichuan University. His research interests include optimal dispatching of distribution networks and machine learning.

Hongjun Gao

Hongjun Gao received the B.S., M.S., and Ph.D. degrees in electrical engineering from Sichuan University, Chengdu, China in 2011, 2014, and 2017, respectively. From 2015 to 2016, he was a Visiting Scholar at the Department of Electrical Engineering and Computer Science, University of Wisconsin-Milwaukee, Milwaukee, WI, USA. He is currently an Associate Professor at the College of Electrical Engineering, Sichuan University, Chengdu, China. His research interests include active distribution system planning and operation, unit commitment, economic dispatch, distributed generation integration, and multi-energy system optimization.

Xiaohui Wang

Xiaohui Wang received his PhD from the North China Electric Power University, Beijing in 2012. He is currently at the China Electric Power Research Institute Co. Ltd., Beijing. His research interests include power big data technology, artificial intelligence, active distributed networks, and the Internet.

Junyong Liu

Junyong Liu received the Ph.D. degree in Electrical Engineering from Brunel University, Uxbridge, U.K., in 1998. He is currently a Professor with the College of Electrical Engineering, Sichuan University, Chengdu, China. He is the director of Sichuan Province Key Smart Grid Laboratory. His current research interests include power system planning, operations, and computer applications.

Kunyu Zuo

Kunyu Zuo (S’21) Received the B.S. and M.S. degrees in electrical engineering from Sichuan University, Chengdu, China, in 2016 and 2019, respectively. From 2017 to 2018, he was a Visiting Student at School of Electronics, Electrical Engineering and Computer Science, Queen’s University Belfast, UK. He is currently pursuing the Ph.D. degree in the Electrical and Computer Engineering Department, Stevens Institute of Technology, Hoboken, NJ, USA. His research interests include renewable microgrid solutions and distributed control.

编辑：刘通明

审核：王伟

郑重声明

根据国家版权局相关规定，纸媒、网站、微博、微信公众号转载、摘编本网站作品，需包含本网站名称、二维码等关键信息，并在文首注明《全球能源互联网》原创。个人请按本网站原文转发、分享。