Why don't the Borg use time travel more often?

In the movie, the Borg tactic of using time travel to take over earth is depicted as a last resort after the mothership is destroyed: An improvised desperate move after more traditional tactics failed.

But why? If time travel is so easy for the Borg, why would it be a last resort? Why wouldn't that be their first option? Going back in time to take over a defenseless earth would be a lot easier then trying to wipe out the Federation with one ship, or even a fleet of ships.

Shouldn't they have just secretly sent back a ship in some unoccupied area of space to go back and prevent first contact? If they did that, the would take over earth easily with zero losses and no one would come back with them to try and stop their efforts.

At the very least, you would think Time travel would be the Borg's default #2 option any time they encounter a race that successfully fights back against them:

"Oh they destroyed one of our cubes? No problem. Send a ship back to take them over before they acquired space-travel technology."