How to filter data with dplyr to retrieve NAs along with another value together?

I have a dataset that looks like this

X      Y      
121    Yes
122    No
123    NA
124    Yes
125    NA

How can I filter out the "Yes" values from column Y ? I only want 'No' and 'NA'

My Desired output is this

X      Y      
122    No
123    NA
125    NA

I've tried

data2 <- data %>% filter(Y != "Yes") #But I lose my NAs with this command.

& 

data2 <- data %>% filter(Y %in% c(is.na(Y), "No")  #Doesn't work

What am i doing wrong?