Two color gradients with geom_raster and geom_point

I have a pretty simple question but I've been struggling to find answers online.

I have a heatmap made with ggplot + geom_raster. I want to insert a symbol inside each rectangle, so they can convey another piece of information. For this, I thought of using geom_point.

The plotting is great but I'm having problems with the color gradients.

plot <- ggplot(data, aes(x = Var1, y = Var2, fill = value)) +
        geom_tile() + 
        geom_raster(aes(fill=value)) +
        scale_fill_gradient2(low = "gray", high = "red", mid = "#e3e3e3", midpoint = "0") + 
        geom_point(data = significance, aes(x = Var1, y = Var, color = value), shape = 21, size = 3) +
        scale_color_gradient(low = "gray", high = "gray")

But, for some reason, the scatterplot is absorbing both gradients, instead of just absorbing the "color" one. Since the heatmap has values very close to zero, and the scatterplot has high values, the "fill" part makes the heatmap all gray.

I used this as a reference to try two gradients: Using two scale colour gradients ggplot2