When I set System Monitor full screen, then close it, next time it relaunches, it will start full screen.

In other words: normal behavior is that it remembers the size of the window at the time it was closed.

In particular, the application sets the dconf setting org.gnome.gnome-system-monitor maximized on true whenever you maximize the window. It stores the current window size of the not maximized window in org.gnome.gnome-system-monitor window-state.

Thus, unmaximize the window, then close the application to have it start with regular window size next time.

Additional caveat: Depending on the settings of the window manager, a large window may be automatically maximized by the window manager. That could explain why you have the impression that it always starts maximized. So make sure you make the window sufficiently small for it not to be maximized automatically. Whether a window that is near the size of the monitor is automatically maximized or not is determined by the dconf setting org.gnome.mutter auto-maximize. You can disable that behavior by setting this to false.

gsettings set org.gnome.mutter auto-maximize false