We do not worry about the latency to the AP unless it is stupidly high, but then again our tower switches would be filling our log servers with lots of complaints then too.
Testing through an AP is almost impossible to use as a comparison to another unless exactly the same number of SMs are connected with exactly the same throughput being used across the SMs. We watch the backhauls, those are more likely where latency issues will be shown.