No no no no no!
You should never do that...never, it's the no 1 rule of travel...never ever ever go inside the homes of strangers, whether cab drivers, friendly locals who start talking to you in malls etc and invite you to their homes ...
Quickest way to get oneself bound, gagged, and hood over head whisked away to a remote location to be held hostage by terrorists and ransomed.
Once you're in a taxi, the driver could just drive you to any home, alley, wherever and there's little you could do about it. Especially since in this case we were driving between towns, but even in cities a few turns could lead you to a desolate area.
You should use some discretion
before getting into a taxi and choose a driver who seems decent and competent. Your best assurance really is that the guy has a legit job and is a presumably licensed driver. [actually I'm fairly amazed by the Uber craze where you just get into some strangers car, whereas in SH I trust more a guy working for a reputable taxi chain].
You always need to use a little discretion and common sense.
For example, I probably never get in a taxi if there is another person, a friend of the driver planning to go along. That's a red flag for me.
In general, in foreign countries, I prefer old guy taxi drivers over young guys who look bigger than me. Less chance of any confrontation. Though young folks more likely to speak some English.
In Georgia, taxi drivers were either real old guys or young tough looking guys with military haircuts and camouflage clothing. We got in one with a burly fellow with a cracked windshield, for a 20 minute drive outside the capital to the main national religious site, and while he had that ex-Soviet tough exterior which put me on mild alert, he turned out to be a real sweetheart and with very limited English pointed out famous sites and such along the way. So you never know. Taxi fares were very low there, which is probably why many had no money to make rather needed repairs (cracked windshield, bumpers missing, etc.).
Otherwise it was interesting to go into a Palestinian home between Bethlehem and Hebron* and see a little how folks lived. And chat about family and kids and such. Our driver's father is also a taxi driver and together they run a small business. While taxi drivers there also willingly act as guides.
* Hebron, the largest city in the West Bank, is the Israeli name for the city. It's called El Khalil in Arabic, which literally means The Friend, referring specifically to Abraham who was God's chosen one, who bought a parcel of land there where he buried his wife Sarah. Al Khalil is short for Khalil al-Rahman (إبراهيم خليل الرحمن) used to refer to Abraham in the Quran, and translated as simply "Friend of God" or more poetically "Beloved of the Merciful." I'm pretty sure that's also the meaning of Hebron in Hebrew.
I think that's a remarkable name for a city -- Friend (of God).
Hebron/El Khalil is also a flashpoint as it's the only West Bank city with Israeli settlers (600 or so) and soldiers and checkpoints within the city limits. Netanyahu just announced expanding the Hebron settlements.