In the waste heat recovery of the internal combustion engine (ICE), the transcritical CO2 power cycle still faces the high operation pressure and difficulty in condensation. To overcome these challenges, CO2 is mixed with organic fluids to form zeotropic mixtures. Thus, in this work, five organic fluids, namely R290, R600a, R600, R601a, and R601, are mixed with CO2. Mixture performance in the waste heat recovery of ICE is evaluated, based on two transcritical power cycles, namely the recuperative cycle and split cycle. The results show that the split cycle always has better performance than the recuperative cycle. Under design conditions, CO2/R290(0.3/0.7) has the best performance in the split cycle. The corresponding net work and cycle efficiency are respectively 21.05 kW and 20.44%. Furthermore, effects of key parameters such as turbine inlet temperature, turbine inlet pressure, and split ratio on the cycle performance are studied. With the increase of turbine inlet temperature, the net works of the recuperative cycle and split cycle firstly increase and then decrease. There exist peak values of net work in both cycles. Meanwhile, the net work of the split cycle firstly increases and then decreases with the increase of the split ratio. Thereafter, with the target of maximizing net work, these key parameters are optimized at different mass fractions of CO2. The optimization results show that CO2/R600 obtains the highest net work of 27.43 kW at the CO2 mass fraction 0.9 in the split cycle.