Just found a simple way to do it.

There is a program called xprintidle that does the trick

getting the idle time (in milliseconds) is as simple as

xprintidle

and to install

apt-get install xprintidle

For sysadmins, it also works remotely

From an ssh session:

export DISPLAY=:0 && sudo -u john xprintidle

where john is the user logged into the X session on the remote machine.

Note that some programs such as MPlayer seem to reset the counter.

That might be desirable or not depending on your application. For me I meant to use it to suspend the computer and the MPlayer exception is helpful.

There is another answer (https://askubuntu.com/a/1303464/56440) for those who want no resetting, but I haven't personally tested it