The Asian ginseng root (Panax ginseng C.A. Meyer) is a very commonly used herbal medicine worldwide. Ginseng fruit, including the berry (or pulp) and seed, is also valuable for several health conditions including immunostimulation and cancer chemoprevention. In this study, the anticancer and anti-proliferative effects of the extracts of ginseng berry and seed were evaluated. The ginsenosides in the ginseng berry concentrate (GBC) and ginseng seed extract (GSE) were analyzed. We then evaluated their anti-colorectal cancer potentials, including antiproliferation, cell cycle arrest, and apoptotic induction. Further investigation consisted of the berry's adaptive immune responses, such as the actions on the differentiation of T helper cells Treg, Th1, and Th17. The major constituents in GBC were ginsenosides Re and Rd, which can be compared to those in the root. The GBC significantly inhibited colon cancer cell growth, and its anti-proliferative effect involved mechanisms including G2/M cell cycle arrest via upregulation of cyclin A and induction of apoptosis via regulation of apoptotic related gene expressions. GBC also downregulated the expressions of pro-inflammatory cytokine genes. For the adaptive immune responses, GBC did not influence Th1 and Treg cell differentiation but significantly inhibited Th17 cell differentiation and thus regulated the balance of Th17/Treg for adaptive immunity. Although no ginsenoside was detected in the GSE, interestingly, it obviously enhanced colon cancer cell proliferation with the underlined details to be determined. Our results suggested that GBC is a promising dietary supplement for cancer chemoprevention and immunomodulation.