The Houthis moved into al-Manasseh area in al-Bayda province under cover from Katyusha rocket fire from the Yemeni army and presidential guard.
Tribal sources said the Ansar al-Sharia fighters withdrew to another district called Bakla, some 3 km (two miles) away.
The Houthis have advanced into central and western Yemen since they seized control of Yemeni capital Sanaa on Sept. 21, taking on Qaeda militants.
Fighting has flared up in several provinces, alarming neighbor Saudi Arabia, the world's No. 1 oil exporter.
The Yemeni army had so far avoided clashing with the Houthis or to support them in their advance on al Qaeda strongholds. But President Abd-Rabbu Mansour Hadi considers Al Qaeda in the Arabian Peninsula (AQAP) as the main threat facing the country, Reuters reported.
One tribal leader said the Houthis exploited old rivalries between two wings of one of the main tribes in the area, the Al al-Dhahab tribe, and managed to enter the area, which had long been the main stronghold of Ansar al-Sharia, the local wing of Al Qaeda in the Arabian Peninsula (AQAP).
Houthi officials made no comment but group's television station said Houthi fighters and their allies have "reached the home of the leader of the criminal gangs" in al-Manasseh, referring to Ansar al-Sharia.