R, ggplot2 - In the legend, how do I hide unused colors from one geom while showing them in others?

I'm making a plot that has color defined for a geom_point(), and everything looks good.

points_a <- data.frame(x = sample(1:10, 4), y = sample(50:60, 4), id = "a")
points_b <- data.frame(x = sample(1:100, 4), y = sample(1:100, 4), id = "b")
points_c <- data.frame(x = sample(1:100, 4), y = sample(1:100, 4), id = "c")
points_all <- rbind(points_a, points_b, points_c)

ggplot(points_all) + geom_point(aes(x, y, color=id))

I'd like to highlight a group of points by drawing a rectangle around them using geom_rect(), picking up the same color already used in the chart. As desired, the legend adds a border around the item in the key; unfortunately, it also adds a border to every other item in the legend, as shown below:

my_box <- data.frame(left = 1, right = 10, bottom = 50, top = 60, id = "a")
ggplot(points_all) + 
  geom_point(aes(x, y, color=id)) + 
  geom_rect(data = my_box, 
    aes(xmin=left, xmax=right, ymin=bottom, ymax=top, color = id), 
    fill = NA, alpha = 1)

I want to get rid of the outlines around items "b" and "c" in the legend, since they're not plotted. I don't know how to do that, since they're defined by the same color aesthetic as the points. Ideally these unused factors should have been dropped from the legend for the outline color, as they rightfully are when showing only one geom_, but that doesn't seem to be how it works. (And defining the color manually outside the aes() call means it wouldn't get shown on the legend for id="a".)

Lots of searching hasn't yet yielded an answer, though I may have overlooked something. What's the best way to hide from the legend unused colors for one geom_ while keeping them for others? (Alternatively: Should I split these into two legends, and how?)