Abstract
Cyber image security is an important issue in current communication networks, as secret images are often communicated through insecure channels. Most existing image encryption algorithms based on conventional S-box designs and chaotic maps are mostly subject to weak cryptanalysis resistance, poor efficiency, or poor flexibility. In order to overcome such limitations, this paper introduces a new substitution box (S-box) design method based on the parametric action of Generalized Triangle Groups (GTG), represented as [Formula: see text]. The new scheme employs GTG-based substitution and permutation operations for RGB image encryption. The research process entails: (i) rigorous mathematical parameterization of GTG to design nonlinear and bijective S-boxes, (ii) thorough analysis of their cryptographic attributes, such as nonlinearity, differential uniformity, strict avalanche criterion, and linear/differential attack resistance, and (iii) integration of the S-boxes in an RGB image encryption system. Experimental testing proves that the suggested S-box has excellent nonlinearity 112, well-balanced Strict Avalanche Criterion ≈0.50, and minimal BIC values, showing very good resistance towards linear attacks, also entropy levels approaching the ideal 8.0, NPCR of more than 99.8%, and UACI of ≈ 33.3%, better than a number of state-of-the-art approaches. In comparison to state-of-the-art S-box and image encryption algorithms, the new scheme has larger nonlinearity, enhanced diffusion/confusion characteristics, and greater cryptanalysis resistance with comparable computational efficiency.