How to replace a value in a binary file in R?

I have to work with a (custom) binary file format. The data are too big for my RAM and I just need to import a small part of them, do some calculations and overwrite/replace this part with the new values (so I don't want to import everything, change specific part and write everything back).

I tried a combination of seek and writeBin but this generates a small file with my new value prepended by zeros:

fn <- tempfile()

writeBin(1L:3L, fn, size = 1L)
readBin(fn, what = "integer", size = 1L, n = 3L)
#> [1] 1 2 3

fh <- file(fn, "wb")
isSeekable(fh)
#> [1] TRUE
seek(fh, 1L, origin = "start", rw = "write")
#> [1] 0
# swap the sign of the second value
writeBin(-2L, fh, size = 1L)
close(fh)

readBin(fn, what = "integer", size = 1L, n = 3L)
#> [1]  0 -2

unlink(fn)

Using the ab mode for appending to the file doesn't help either:

fn <- tempfile()

writeBin(1L:3L, fn, size = 1L)
readBin(fn, what = "integer", size = 1L, n = 3L)
#> [1] 1 2 3

fh <- file(fn, "ab")
isSeekable(fh)
#> [1] TRUE
seek(fh, 1L, origin = "start", rw = "write")
#> [1] 3
# swap the sign of the second value
writeBin(-2L, fh, size = 1L)
close(fh)

readBin(fn, what = "integer", size = 1L, n = 3L)
#> [1] 1 2 3

unlink(fn)

My expected output would be 1 -2 3.

Is there a way to do this in R or do I have to use C for it?