In PVE if it is going to take you X time to cast/use some skill, that is what is presented. It doesn't work correctly in PVP, I have had several instances when the time listed is less than the action bar for the opponent, but his attack goes off first.
times listed on actions- Haste Buff/ Debuff Expires?
I thought that at one point and then realized that what was happening was slow/haste effects where timing out mid action and thus changing the time it takes to do said action. Are you sure that's not what's happening to you?
Could be - I will have to pay attention to that part of the equation.