Docetaxel (DTX) is one of the most important anticancer drugs; however, the severity of its adverse effects detracts from its practical use in the clinic. Magnetic nanoparticles of Fe3O4 (MgNPs-Fe3O4) can enhance the delivery and efficacy of anticancer drugs. We investigated the effects of MgNPs-Fe3O4 or DTX alone, and in combination with prostate cancer cell growth in vitro, as well as with the mechanism underlying the cytotoxic effects. MgNPs-Fe3O4 caused dose-dependent increases in reactive oxygen species levels in DU145, PC-3, and LNCaP cells; 8-hydroxydeoxyguanosine levels were also elevated. MgNPs-Fe3O4 alone reduced the viability of LNCaP and PC-3 cells; however, MgNPs-Fe3O4 enhanced the cytotoxic effect of a low dose of DTX in all three cell lines. MgNPs-Fe3O4 also augmented the percentage of DU145 cells undergoing apoptosis following treatment with low dose DTX. Expression of nuclear transcription factor ?B in DU145 was not affected by MgNPs-Fe3O4 or DTX alone; however, combined treatment suppressed nuclear transcription factor ?B expression. These findings offer the possibility that MgNPs-Fe3O4-low dose DTX combination therapy may be effective in treating prostate cancer with limited adverse effects.