Geothermal energy, a renewable energy source with enormous reserves independent of the external environment, is essential for reducing carbon emissions. Spiral fin coaxial borehole heat exchanger (SFCBHE) is vital for geothermal energy extraction. Its heat extraction performance requires further improvements for efficient performance that consider the structural sizes and installation positions of the SFCBHE and the nonlinear coupling with respect to several factors. The heat extraction performance of SFCBHE is optimized using a combination of genetic algorithm–back-propagation neural network (GA–BPNN) and the Q-learning-based marine predator algorithm (QLMPA). This study analyzes and compares the effects of geothermal energy extraction of smooth pipe TY-1, structure before optimization TY-2, and optimized structure TY-3. Following optimization with GA–BPNN–QLMPA, the heat extraction performance of TY-3 is enhanced by 30.8% and 23.6%, respectively. The temperature of maximum extraction is improved by 26.8 K and 24.0 K, respectively. The power of maximum heat extraction is increased by 148.2% and 109.5%, respectively. The optimization method can quickly and accurately determine the heat extraction performance for different structural sizes and installation positions of the SFCBHE. These findings are crucial for developing high-performance SFCBHE and efficiently using geothermal energy.