Using fill in ggplot to colour polygons and boxplots with different types of scale

I'm trying to great a plot with a grouped boxplot over the top of a set of polygons, and would like to use the fill aesthetic to colour the polygons based on a continuous variable, and the boxplot based on a separate grouping variable. I've found several posts that helped me get the boxplot over the polygons, and have found someone else who had a related problem, but nothing with a solution.

This is very nearly the type of plot I want:

rectangle <- data.frame(x = c("Fair", "Very Good", "Fair", "Very Good"),
lower = c(rep(3000, 2), rep(5500, 2)),
upper = c(rep(5000, 2), rep(7000, 2)),
band = c(1,1,2,2))
ggplot() + geom_blank(data=diamonds, aes(x=cut, y=price, colour = color)) +
geom_rect(data=rectangle, aes(xmin=-Inf, xmax=Inf, 
                                    ymin=lower, ymax=upper, fill = band), alpha=0.1) +
geom_boxplot(data=diamonds, aes(x=cut, y=price, colour = color)) 

But I'd like to use 'fill' instead of 'colour' to do the boxplots, as it will work much better in the context I actually need it. If I use fill in both the geom_rect and the geom_boxplot I end up with an error:

ggplot() + geom_blank(data=diamonds, aes(x=cut, y=price, colour = color)) +
geom_rect(data=rectangle, aes(xmin=-Inf, xmax=Inf, 
                                ymin=lower, ymax=upper, fill = band), alpha=0.1) +
geom_boxplot(data=diamonds, aes(x=cut, y=price, fill = color))

##Error: Discrete value supplied to continuous scale 

Is there a way I can complete this figure they way I need? Thanks!