Could you provide your performance reports and your XMT configuration (screenshot)?
It's been tested via P3 with a promising result (XMT takes less than 0.025s).
Here is the P3 report of XMT and some other plugins performance:
You can also know the plugin performance by adding ?xmt_debug_show at the end of your URL.
As you can see, XMT costs 0s (means less than 0.5s so that it's rounded to 0s)
Here is the debug result: