Replace cell values, match list elements and switch the pair

I have a list of duplicates l1 <- list(c("a","b"), c("1","2"),c("x","y")). There is also a dataframe as such:

     df <- structure(list(names = structure(c(2L, 1L, 3L, 4L), .Label = c("1", 
"a", "x", "y"), class = "factor"), values = c(0, 0, 0, 0)), class = "data.frame", row.names = c(NA, 
-4L))

  names values
1     a      0
2     1      0
3     x      0
4     y      0

I would like to replace any occurrences of these names with its pair from the l1 list. The desired output with df would be:

  names values
1     b      0
2     2      0
3     y      0
4     x      0