I've been having this problem with VF4 VanillaCraft, and decided to try it on another server... turns out it's an issue with every server I've tried to join.
When i play on singleplayer (survival and creative), I have no issues. As soon as I get on multiplayer, it won't even load ten blocks...