Help with Voip issue

I am using Voip for phone/network traffic between buildings. All of my buildings work fine except 1. The power levels and jitter are comparable at all sites. I am using 20mb BH’s (5.2) from our tower to our main building where the main telephone system is. I am using 5.7 SM’s at all sites. The software version is 6.1. Is there a way to increase the efficiency of the Voip traffic to and from the building causing the problem? It seems that there is a delay in the 1st 10-15 seconds of the call and then it gets better, but the volume level of the calls are always low. There is no other significant network traffic at that building, either. Thanks