Custom axis breaks in ggplot2 facets based on other columns

I have data to plot, where the x-axis is found in one column, and the x-axis major breaks are found in other columns.

For my sample data, I will modify the iris dataset from ggplot2. Note: the low and high are arbitrarily calculated here - I've only chosen min & max for ease of reproducibility.

library(dplyr)
library(ggplot2)


df <- iris %>% 
  group_by(Species) %>% 
  mutate(low = min(Sepal.Length),
         high = max(Sepal.Length)) %>% 
  ungroup()

> df
# A tibble: 150 x 7
   Sepal.Length Sepal.Width Petal.Length Petal.Width Species   low  high
          <dbl>       <dbl>        <dbl>       <dbl> <fct>   <dbl> <dbl>
 1          5.1         3.5          1.4         0.2 setosa    4.3   5.8
 2          4.9         3            1.4         0.2 setosa    4.3   5.8
 3          4.7         3.2          1.3         0.2 setosa    4.3   5.8
 4          4.6         3.1          1.5         0.2 setosa    4.3   5.8
 5          5           3.6          1.4         0.2 setosa    4.3   5.8
 6          5.4         3.9          1.7         0.4 setosa    4.3   5.8
 7          4.6         3.4          1.4         0.3 setosa    4.3   5.8
 8          5           3.4          1.5         0.2 setosa    4.3   5.8
 9          4.4         2.9          1.4         0.2 setosa    4.3   5.8
10          4.9         3.1          1.5         0.1 setosa    4.3   5.8
# ... with 140 more rows

I am hoping to plot x = Sepal.Length faceting for Species, but with the only two major breaks being df$min and df$max.

I'm having trouble getting the breaks in the correct facet.

df %>% 
  ggplot(aes(x = Sepal.Length,
             y = Petal.Length)) + 
  geom_point() + 
  facet_wrap(. ~ Species) + 
  scale_x_continuous(breaks = c(df$low, df$high))

As you can see, the values from df$low and df$high applied to all facets. I was hoping that facet setosa would only have major breaks at 4.3 and 5.8 only, versicolor at 4.9 and 7.0 only, and virginica at 4.9 and 7.9 only.

Is there a way to pass the facet variable to breaks in scale_x_continuous? Or should I abandon this approach and create three separate ggplots and merge them together with gridExtra?

Any help would be appreciated!