Yes, that's how I play it. It was a rumored that they were there and you missed them. They heard you were coming and ran away before you got there (maybe to setup an ambush somewhere else). Or make up some other reason that would drive your story.
Diplomacy is the weapon of the civilized warrior. - A.T. Hun -- Robert Aspirin