adding points to raster plots with facets

I have a few raster plots separated with facets. In each plot, I want to add an independent point. This shows how to add a point but I can only add the same point to all plots.

Suppose I want to add a point at the maximum value of the following three plots (the code is given below). How can I do that?

xy    <- expand.grid(0:20,0:20)
data  <- rbind(xy,xy,xy)
group <- rep(1:3,each=nrow(xy))
set.seed(100)
z     <- rnorm(nrow(data))
data <- cbind(data,group,z)
colnames(data) <- c("x","y","group","z")
library(ggplot2)
ggplot(data,aes(x,y,z))+geom_raster(aes(fill=z))+facet_wrap(~group)