Zechu (Steven) Li

I am currently a research assistant at MIT CSAIL, advised by Prof. Pulkit Agrawal. My research interest lies in reinforcement learning, especially its high-performance and scalable systems and applications (e.g., robotics, finance, and transportation).

Prior to this, I received my bachelor's degree from Columbia University in May 2022, majoring in computer science. During my undergraduate studies, I was fortunate to work with Prof. Xiaodong Wang, Prof. Anwar Walid and Prof. Sharon (Xuan) Di.

Email  /  Google Scholar  /  GitHub /  LinkedIn

Autonomous driving (AV) technology has elicited discussion on social dilemmas where trade-offs between individual preferences, social norms, and collective interests may impact road safety and efficiency. In this study, we aim to identify whether social dilemmas exist in AVs’ sequential decision making, which we call “sequential driving dilemmas” (SDDs), to help policymakers and AV manufacturers better understand under what circumstances SDDs arise and how to design rewards that incentivize AVs to avoid SDDs.

We present a novel parallel Q-learning framework that not only gains better sample efficiency but also reduces the training wall-clock time compared to PPO. Different from prior works on distributed off-policy learning, such as Apex, our framework is designed specifically for massively parallel GPU-based simulation and optimized to work on a single workstation. We demonstrate the capability of scaling up Q-learning methods to tens of thousands of parallel environments.

We propose a quantum K-spin Hamiltonian regularization term (called H-term) to help a policy network converge to a high-quality local minima. We take a quantum perspective by modeling a policy as a K-spin Ising model and employ a Hamiltonian equation to measure the energy of a policy. We derive a novel Hamiltonian policy gradient theorem and design a generic actor-critic algorithm that utilizes the H-term to regularize the policy network. The proposed method significantly reduces the variance of cumulative rewards by 65.2% ~ 85.6% on six MuJoCo tasks, etc.

We propose a homomorphic matrix completion algorithm for privacy-preserving purpose. We first formulate a homomorphic matrix completion problem where a server performs matrix completion on cyphertexts, and propose an encryption scheme that is fast and easy to implement. We prove that the proposed scheme satisfies the homomorphism property and satisfies the differential privacy property. While with similar level of privacy guarantee, we reduce the best-known error bound to EXACT recovery at a price of more samples.

We apply the social learning scheme to Markov games and leverage deep reinforcement learning (DRL) to investigate how individual AVs learn policies and form social norms in traffic scenarios. To capture agents' different attitudes toward traffic environments, a heterogeneous agent pool with cooperative and defective AVs is introduced to the social learning scheme. To solve social norms formed by AVs, we propose a DRL algorithm, and apply them to traffic scenarios: unsignalized intersection and highway platoon.

We introduce an RLOps in finance paradigm and present a FinRL-Podracer framework to accelerate the development pipeline of deep reinforcement learning (DRL)-driven trading strategy and to improve both trading performance and training efficiency. We evaluate the FinRL-Podracer framework for a stock trend prediction task on an NVIDIA DGX SuperPOD cloud, and show the high scalability by training a trading agent in 10 minutes with 80 A100 GPUs, on NASDAQ-100 constituent stocks with minute-level data over 10 years.

We present a scalable and elastic library ElegantRL-podracer for cloud-native deep reinforcement learning, which efficiently supports millions of GPU cores to carry out massively parallel training at multiple levels. At a high-level, ElegantRL-podracer employs a tournament-based ensemble scheme to orchestrate the training process on hundreds or even thousands of GPUs, scheduling the interactions between a leaderboard and a training pool with hundreds of pods. At a low-level, each pod simulates agent-environment interactions in parallel by fully utilizing nearly 7,000 GPU CUDA cores in a single GPU.

FinRL is the first open-source framework to show the great potential of financial reinforcement learning. It has evolving into an ecosystem, including hundreds of financial markets, state-of-the-art algorithms, financial applications (portfolio allocation, cryptocurrency trading, high-frequency trading), live trading, cloud deployment, etc.

ElegantRL is a massively parallel library for cloud-native deep reinforcement learning (DRL) applications. Our mission is to provide a scalable, efficienct, and accessible DRL platform for researchers and practitioners to develop cutting-edge RL applications. My specific interests are the development and usage of massively parallel simulations, and large-scale training with population-based training, ensemble methods, etc.

As a leader of this project, I have been contributing to

• develop a series of large-scale training frameworks,
• implemente SOTA algorithms and techniques,
• build the documentation website.

• ElegantRL: Much More Stable Deep Reinforcement Learning Algorithms than Stable-Baseline3,
  MLearning.ai, Mar. 3, 2022.
• ElegantRL-Podracer: A Scalable and Elastic Library for Cloud-Native Deep Reinforcement Learning,
  Towards data science, Dec. 11, 2021.
• ElegantRL: Mastering PPO Algorithms,
  Towards data science, May. 3, 2021.
• ElegantRL Demo: Stock Trading Using DDPG (Part II),
  MLearning.ai, Apr. 19, 2021.
• ElegantRL Demo: Stock Trading Using DDPG (Part I),
  MLearning.ai, Mar. 28, 2021.
• ElegantRL-Helloworld: A Lightweight and Stable Deep Reinforcement Learning Library,
  Towards data science, Mar. 4, 2021.

Large-scale deep neural networks (DNNs) have led to impressive successes in many applications. However, two challenges often arise in DNN deployment in Internet of Things (IoT) devices and real-time applications: training time and memory footprint. This chapter takes a practical approach to seek a better efficiency-accuracy trade-off, which utilizes high performance tensor decompositions to compress and accelerate neural networks by exploiting low-rank structures of the network weight matrix.

This guy makes a nice webpage.