Abstract
The exploiting of hybrid beamforming (HBF) in massive multiple-input multiple-output (MIMO) systems can enhance the system's sum rate while reducing power consumption and hardware costs. However, designing an effective hybrid beamformer is challenging, and interference between multiple users can negatively impact system performance. In this paper, we develop a scheme called Subset Optimization Algorithm-Hybrid Beamforming (SOA-HBF) that is based on the subset optimization algorithm (SOA), which effectively reduces inter-user interference by dividing the users set into subsets while optimizing the hybrid beamformer to maximize system capacity. To validate the proposed scheme, we constructed a system model that incorporates an intelligent reflecting surface (IRS) to address obstacles between the base station (BS) and the users set, enabling efficient wireless communication. Simulation results indicate that the proposed scheme outperforms the baseline by approximately 8.1% to 59.1% under identical system settings. Furthermore, the proposed scheme was applied to a classical BS-users set link without obstacles; the results show its effectiveness in both mmWave massive MIMO and IRS-assisted fully connected hybrid beamforming systems.