Those tiny border towns don't really enforce it, but if you want to travel further along the border, or if you're crossing at any significant port of entry, you need it. I'd often cross at Calais, Maine. Sometimes, I'd take the ferry from Bar Harbor or even Portland. For that, you'd definitely need a passport. But the border towns... small ones? It depends on the precise situation.