Not sure how it works, but I accidentally clicked "+" on LoopEnd and that fixed the problem! Looks like Sampler counts sample length differently or something.
My loop end point is exactly at the end of the sample so I shouldn't be able to increase LoopEnd position by clicking "+", but all samples have +1 "gap" at the end of the sample.
And if the loop region is in the middle of the sample, then "+1" or "-1" on LoopEnd or LoopStart fixes the problem.
Maybe this will be useful for someone :)