To address the issue that the stage–discharge relationship in natural watercourses exhibits a non-monotonic relationship due to the combined effects of variable backwater, rising and falling limbs of floods, and their interaction, a hybrid PSO-NLS calibration model for the stage–discharge relationship was developed based on the physical coupling of water flow. The model uses cross-sectional area and effective head upstream and downstream as the basic framework for water conveyance, incorporating backwater water level thresholds, backwater intensity, and water level fluctuation rates to characterize downstream backwater, backwater intensity modulation, and flood fluctuation hysteresis effects, respectively. Parameters are globally searched using the Particle Swarm Optimization (PSO) algorithm and locally refined using multi-start Nonlinear Least Squares (NLS). Using the Shaxiping Station as a case study and based on observed water level and discharge data from 2020–2021, four model types were proposed and compared: water level–head, effective head, backwater intensity correction, and fully coupled models. The results show that for the stage–discharge relationships in 2020 and 2021, the complete coupling model achieved determination coefficients of 0.9983 and 0.9993, respectively, with RMSEs of 22.44 m3·s?1 and 7.39 m3·s?1, and MAEs of 3.95 m3·s?1 and 0.88 m3·s?1, respectively. The model comparison indicates that the area index, effective head threshold, and comprehensive conveyance coefficient have a significant impact on the calculation results and are key parameters for calibration. The research findings effectively reduce the variability of the stage–discharge relationship under complex hydraulic conditions and can serve as a reference for discharge estimation in watercourses affected by downstream backwater and flood fluctuations.