Converting dates in column of a data frame R

I am having problems converting a column of imported dates in a data frame, represented as characters in a different date format, into date objects in that same data frame. Here is a toy example:

xx <- data.frame(A = c(10, 15, 20), B = c("10/15/2010", "9/8/2015", "8/5/2013"))

If I print xx,

   A          B
1 10 10/15/2010
2 15   9/8/2015
3 20   8/5/2013

I apply:

xx[, "B"] <- sapply(xx[, "B"], function(x) {as.Date(x, 
           format = "%m/%d/%Y", origin = "1970-01-01")})

and I get:

   A     B
1 10 14897
2 15 16686
3 20 15922

If I look at the mode of column B, it is numeric, not date. No matter what I try I cannot seem to get a result that converts column B to a date type. I can always add:

xx[, "B"] <- as.Date(xx[, "B"]) 

but there must be a way to do this in one statement.