You can find the operating system breakdown, amongst other statistics for the top 500 supercomputers here.

About 90% use some variety of Linux. None report using Ubuntu, but almost none specify exactly which Linux distribution they use.

I suspect the majority use their own customised distributions with both the kernel and software tailored for their hardware and major uses. My guess would be that they would use either Red Hat or Debian (as Ubuntu does) as a starting point for customisation, but this is just my supposition.