Ubuntu's community documentation explains the real reasons why it's not recommended to run a GUI on a production server system:

Most Ubuntu Server developers do not recommend installing X on a
server. There are multiple reasons for not installing a GUI.

Some reasons to not install a GUI include:

1. You'll have more code subject to security vulnerabilities, more packages that need updating, and more server downtime.


2. X11 and desktop packages are not supported for the full 5 year lifecycle of the LTS server release.


3. Performance may suffer because resources (memory, hard disk space, CPU, etc.) will be consumed by the GUI.


4. It is best practice to only install needed software on a production server.


5. The GUI may include other network services that are inappropriate for a server.
   
   
   
   1. One of the goals of Ubuntu Desktop Edition is to make it easier for users to use Linux. When installing some desktop environments,
      services that you may not specifically want will be installed. For
      example avahi-daemon, which is used to help configure networking,
      adds another open port and may introduce unwanted DNS conflicts with a
      .local domain.
   
   

So for the most secure server it is best to not install a GUI.

^{"ServerGUI" by "Contributors to the Ubuntu documentation wiki", reproduced here as permitted by CC-BY-SA 3.0.}

Contrary to somewhat common misconception, X11 being a server really has nothing to do with why running a GUI on a production server is considered to be non-ideal from a security perspective. X11 is virtually never configured by default to be accessible over a network anymore, on any operating system. No version of Ubuntu has ever had X11 run a network-accessible server in the default configuration. (To access X11 on Ubuntu via TCP, you have to forward it though SSH or manually reconfigure the server.)

Furthermore, even if X11 did run a network-accessible server, this would not be a reason not to have it installed on a production server system. Anyone running a production server is presumably capable of configuring it for their needs and auditing it to make sure undesirable services are not running. (If they cannot, that will pose a far greater threat to their security than would be created by having a GUI installed.) Even if X11 had to have a port listening on a physical network interface (which is not the case), the port could easily be blocked by reconfiguring the built-in netfilter using iptables (or a higher-level frontend like ufw).

In contrast, the problems listed above are not so easy to overcome by reconfiguration.