R: Replace rare values by "others"

I have the following problem: I have a data frame df with many variables. One variable is df$size (non-numeric). Now I want to replace all sizes with less than 20 observations by the term "other".

sort(table(df$size))

This gives me an overview of the values I want to replace. But how do I replace them in my df?

df$size[sort(table(df$size))<20]="other"

That does not work.

Thank you!