I have been trying to figure out why rsync was Segfault-ing when trying to transfer several hundred gigabytes from the old server to the new one.

It was working fine but I interrupted it a few times. And then, when I was down to 5-6 gigabytes it would transfer a file or two (small files at that) and just spit out a weird message, like

rsync error: errors with program diagnostics

or it would complain about a timeout (I am copying from an NFS volume, what timeout?), or simply receive a Segfault.

In a desperate attempt to figure out at least when the problem was occurring I decided to rebuild it with debugging symbols enabled.

env CFLAGS=-ggdb3 emerge -av rsync

I then fired up gdb an ran the problematic commands

gdb
run -avPAH /remote/... /var/....

and it worked fine. Since I had -Os in my CFLAGS, I tried using -O2 instead and still, there were no problems.

Lesson learned, -Os does break stuff.

Update: Well, -O2 saved the day for a while but rsync again started segfaulting when I tried to copy a directory with 16000 files (well, 8000 files each with an extra hard link). So I compiled it with no optimization, or better said, with no specific optimization and it worked. For how long I wonder.

Update: I was obviously chasing ghosts. Here is a follow up.