If you are into self-compiling everything, maybe Gentoo is a better choice for you.

Having worked for a Linux distributor for 16 years (and counting), I can assure you that it's far from trivial to compile a complete Linux distribution. The business value of the companies that are doing that is in the know-how, infrastructure and work procedures (and their automation) to do that. This is not anything to easily replicate at home.

You can build one or a handful of packages at home easily, but when it comes to a full distribution, it's a massive task. How many man-years are you willing to spend?

Having said that, Gentoo Linux is based around the idea to let every user do exactly that at home; but still, you need to have some trust in those who provide the source tarballs and all the patches on top of them. If you are suspicious, you can dig deep into one or a few of them and check what they are doing, but it's unrealistic to do that for any meaningful percentage of the whole distribution.

Also, I have doubts that you will manage to optimize the resulting binaries better (at least in any meaningful way) than those who have been doing that for a living for many, many years.

Still, if you feel you must do it, go ahead. ;-)