Conditional cbind() on dates

Assume we have 2 dataframes of 2 cols and 6 rows each and we want to cbind both only when date on the left(lhs) is older than date on the right(rhs), at the same time making sure that each row has no date duplicates (in both lhs and rhs): for instance..

x = cbind(data.frame(lhs_date = seq(Sys.Date()-5, Sys.Date(),2)), letter=c("A","B","C","D","E","F") )
Y = cbind(data.frame(rhs_date = seq(Sys.Date()-5, Sys.Date(),1)), letter=c("X","Y","Y","X","J","J") )

How can we cbind or left join x to y only when lhs date < rhs date maintaining the uniqueness of each row?