The women objected to the Fox News Channel host saying that "Muslims killed us on 9/11." They returned after an O'Reilly apology.
The fireworks came amid a heated argument on the issue.
"Listen to me," O'Reilly said, "because you'll learn something."
"Pinhead!" retorted Behar, who also hosts a prime-time show on HLN.
O'Reilly, who was on the popular daytime talk show to promote his book "Pinheads and Patriots: Where You Stand in the Age of Obama," said locating the Islamic Center near ground zero is inappropriate "because Muslims killed us on 9/11."
Goldberg responded with an expletive and Behar rose from her seat.
"I don't want to sit here," Behar said. "I don't. I'm outraged by that statement."
She walked off the set, followed by Goldberg.
The show's creator, Barbara Walters, immediately said her colleagues were wrong to walk off.
"We should be able to have discussions without washing our hands and screaming and walking offstage," she said. But she also scolded O'Reilly, saying that it was extremists who committed the terrorist act. "You cannot take a whole religion and demean them," she said.