Ggplot2 change the position of y axis values after removing gridlines

I want to create a heatmap in ggplot2 without the gridlines. I removed them using the theme function and panel.grid arguments, but it leaves the space between the y axis values and the plot:

I tried to close this gap using axis.text argument of theme function, but it makes the values not aligned:

Can someone please suggest a solution that would make the plot look like this:

Below the reproducible code:

library(ggplot2)
tmp<-data.frame(cbind(rep(c("Sun", "Sat", "Fri", "Thu", "Wed", "Tue", "Mon"), 24),
                      rep(0:23, 7),
                      runif(168, 0, 3)))
tmp$X1<-factor(tmp$X1, levels= c("Sun", "Sat", "Fri", "Thu", "Wed", "Tue", "Mon"))
tmp$X2<-as.numeric(tmp$X2)
tmp$X3<-as.numeric(tmp$X3)
ggplot(tmp, aes(X2, X1, fill=X3, label=round(X3,2))) +
  geom_tile() +
  theme_minimal() +
  scale_fill_gradient(low="springgreen", high="darkgreen") +
  scale_x_continuous(breaks=0:23) +
  geom_text() +
  theme(panel.grid.major.x = element_blank(),
        panel.grid.major.y = element_blank(),
        panel.grid.minor.x = element_blank(),
        panel.grid.minor.y = element_blank(),
        axis.title.x = element_blank(),
        axis.title.y = element_blank(),
        #axis.text.y = element_text(hjust = 3.2),
        legend.position = "none")