The impressive variation in floral traits among populations and species of flowering plants has raised the long-standing question how such variation evolves and is maintained. The tight linkage between floral traits and reproductive success suggests that trait differences evolve during adaption to contrasting ecological conditions. Pollinators are the usual suspects for drivers of floral trait evolution, but contributions of other biotic and abiotic factors deserve further study. Here we explore how biotic and abiotic factors interact to shape floral trait variation in experimental populations of a wild carnation, Dianthus carthusianorum, originating from high and low elevations in the Swiss Alps and differing markedly in multiple floral traits. Alpine areas are ideally suited for such studies because environmental conditions change sharply with altitude on a small geographic scale. We reciprocally transplant 1'152 seedlings from three high and three low altitude populations into two sites located in the same altitudinal belts, one at high (> 2000 m.a.sl.) and at low (< 1000 m.a.sl.) elevation. A phenotypic selection analysis using seed set as female fitness estimate was performed at the two transplant sites. Our results indicate that environmental influence on selection is trait-specific and tends to be stronger on traits related to plant phenology and architecture instead of pollinator fit. These analyses estimate phenotypic selection exclusively through female fitness and ignore that traits may also evolve through selection on male fitness. To test how phenotypic selection through male function affects selection on flowering traits we performed a parentage analysis including more than 3'000 seeds collected at the low elevation transplant site and estimated siring success for all hermaphroditic plants. We discuss how selection through female and male fitness interaction with the environment to shape variation in flowering-related traits in this alpine herb.