The party's highest decision making body resolved to recall him, invoking the clause in the African National Congress constitution that gives the structure the powers to recall a public representative.
The party was expected to write to Zuma on Tuesday morning, informing him of its decision.
LIVE: ANC to brief media at noon on outcomes of NEC meeting
The ANC NEC held a 13-hour-long marathon meeting in Irene, outside Pretoria. The urgently called meeting, started at 14:00 on Monday and ended at 03:00 on Tuesday.
At 23:00, the NEC instructed party President Cyril Ramaphosa and secretary-general Ace Magashule to drive to Zuma's official residence Mahlamba Ndlopfu in Pretoria to give him an ultimatum to resign or face a recall.
That meeting lasted less than an hour, with Zuma refusing to comply.
Ramaphosa and Magashule delivered that news to the NEC meeting at 00:00. The venue was on lockdown in the hour that the two left to meet Zuma and returned.
NEC sources told News24 that during the trio's meeting, Zuma was still asking "what he did wrong".
"Zuma told Ramaphosa and Magashule that he will respond to them publicly when the party informs him of the decision to recall him," a source said.
The ANC is expected to brief the media on the outcomes on the NEC meeting at 12:00 on Tuesday.
Zuma is said to have demanded to stay on at the Union Buildings for another three months to attend at least two upcoming events. The NEC, however, rejected his conditions.
Sources insisted that Zuma wanted to still be head of state when the Brics summit is held and he wanted attend another African Union meeting.
The Brics summit is scheduled for July in Sandton, which would mean Zuma would stay for five months. Zuma pioneered South Africa's entry to join the Brazil, Russia, India and China grouping.
"We couldn't allow him to stay on. He was disrespecting the organisation. The party also needs to move on, we need to have State of the Nation, we need investors to come back," a source said.
State Capture inquiry
Another NEC member said that Ramaphosa had earlier told the NEC that Zuma had agreed to step down during his direct talks with him last Tuesday, February 6.
At the time Ramaphosa cancelled an emergency urgent NEC meeting that was called for last week Wednesday in Cape Town, describing his talks with Zuma as "fruitful and constructive".
ANC senior member Enoch Godongwana is said to have led the charge for Zuma to go.
He is said to have argued that in light of the State Capture inquiry investigating Zuma's relationship with the Guptas, threat of impeachment and a motion of no confidence against him, recalling him was "saving Zuma from himself".
However, Zuma is said to have been defended by several members, including Deputy Communications Minister Tandi Mahambehlala, ANC Women's league secretary general Meokgo Matuba and North West Premier Supra Mahumapelo.
They are said to have argued that the NEC never gave the top six officials a mandate to recall Zuma but instead to negotiate a transition.
Sources said they also argued that "Everyone in the movement did not agree with a recall."
Another asked: "What was the rush?"
The sources who spoke to News24 on condition of anonymity, could not say if the meeting had decided on the next step if Zuma refused to resign.
One said the party would have to "consider its legal options" should Zuma defy a party directive, while another said Zuma could not deny a party directive.
The most obvious option left for the ANC should Zuma decide not follow the party's directive for his recall would be to vote him out of office in a motion of no confidence.
The ANC is under pressure to decide on Zuma's fate as opposition parties have demanded that Speaker Baleka Mbete move forward a motion of no confidence tabled by the Economic Freedom Fighters.
The EFF has given Mbete until 10:00 on Tuesday to decide on a new date for the motion.
Earlier, News24 reported that if Zuma had refused to resign even after a decision to recall him, the battleground would move to Parliament.
In the past, the ANC had called it the "worst form of betrayal" of the party if its MPs voted with the opposition.
KEEP UPDATED on the latest news by subscribing to our FREE newsletter.
- FOLLOW News24 on Twitter