Calcium sulfate whiskers (CSWs) were hydroxylated with a sodium hydroxide (NaOH) solution and isolated for subsequent treatment with an ethanolic 3-(methacryloxy)propyltrimethoxysilane (KH570) solution to introduce C=C double bonds on the CSWs' surfaces. Then, CSW-g-PMMA was prepared by grafting polymethyl methacrylate (PMMA) onto the surface of modified CSW using in situ dispersion polymerization. The CSW-g-PMMA was used as a filler and melt-blended with polyvinyl chloride (PVC) to prepare PVC-based composites. The surface chemical structure, PMMA grafting rate, and hydrophobic properties of CSW-g-PMMA were analyzed using X-ray diffraction, diffuse reflectance Fourier-transform infrared spectroscopy, thermogravimetric analysis, and water contact angle measurements, respectively. The effects of the CSW-g-PMMA filler on the mechanical properties of the CSW-PMMA/PVC composites were also investigated. The results showed that NaOH treatment significantly increased the number of hydroxyl groups on the surface of the CSWs, which facilitated the introduction of KH570. PMMA was successfully grafted onto the KH570 with a grafting rate of 14.48% onto the surface of the CSWs. The CSW-g-PMMA had good interfacial compatibility and adhesion properties with the PVC matrix. The tensile, flexural, and impact strengths of the CSW-g-PMMA/PVC composite reached 39.28 MPa, 45.69 MPa, and 7.05 kJ/m2, respectively, which were 38.55%, 30.99%, and 20.10% higher than those of the CSW/PVC composite and 54.52%, 40.80%, and 32.52% higher than those of pure PVC, respectively. This work provides a new method for surface modification of inorganic fillers, resource utilization, and high value-added application of CSWs from phosphogypsum.