Count the number of values between value and value - x by variable

I have some data:

library(data.table)
set.seed(1)
df1 <- data.frame(let=sample(sample(letters,2),5, replace=TRUE),
                  num=sample(1:10,5))
setDT(df1)
   let num
1:   j   7
2:   j   6
3:   g   1
4:   j   2
5:   j  10

and I would like to calculate the number of num that are less than or equal to num AND are greater than or equal to num - 4, by let. Using the data.table package would be preferable, but any solution using dplyr or base r would be fine too. The output would look like this:

   let num countNumByLet
1:   j   7             2
2:   j   6             2
3:   g   1             1
4:   j   2             1
5:   j  10             3