Ping times vary at first?

We are using two link of PTP49400 full. We test the bandwidth using IPERF. With the link isolated, and a laptop on each end, I run a ping to the other laptop. These pings are anywhere from 3ms to 200ms. I run the test and the bandwidth is there just fine. After I run the test the ping times are a normal and steady 12ms. I can only replicate the problem by unplugging and plugging the cable back into the laptop and starting the ping again. Any suggestions?

These links are all running together with a midpt in the middle, where the slave of link one and the master of link two are colocated.