Filter only EQUAL values into columns with dplyr::filter_if

I have following data:

df <- data.frame(
  x = c(1, 4, 3, 4, 4, 3), 
  y = c(2, 3, 4, 4, 2, 3)
)

I try use this code:

library(tidyverse)

df %>% 
  filter_if(~ is.numeric(.), all_vars(. %in% c('3', '4')))

  x y
1 4 3
2 3 4
3 4 4
4 3 3

But, the expected result is:

  x y
1 3 3
2 4 4

How do this?