I think it should not cause serious performance issues for "normal" use, but as already pointed out some graphics programs need to scan the fonts at startup (because they need more info from the fonts than the fontconfig database provides) and those will start significantly slower, of course. Also, when fontconfig updates its database (e.g. when you add additional fonts), that might be slower too.

And finding a particular font in the list of available fonts might become a human performance issue... ;)

So it's probably best to use a font management tool like font-manager or fontmatrix, so that you can enable/disable individual fonts or groups of fonts.

BTW, you can count the number of currently installed fonts using the following command:

$ fc-list | wc -l

2083

(Those 2000+ fonts aren't an issue on my system...)