In Istanbul's old city, a groan went through a gathering of hundreds of people.
In the first round, Istanbul and Madrid tied with 26 votes each. Tokyo had 42 votes, six short of a winning a majority. Istanbul then beat Madrid 49-45 in a tiebreak to advance to the final, which Tokyo won easily.
After Madrid lost the tiebreak, a deathly hush fell over a crowd that had assembled in the Spanish capital's Puerta de Alcala square and the music stopped.
"I am in shock," said Marta Castro, a housewife in the square. "I thought that it was a tiebreaker to see which city won and it turns out that it was to see which lost, and Madrid went out first. How sad! I hadn't imagined it."
In their final presentations, Madrid made its case as the least-expensive option and Istanbul spoke of the historic opportunity to bring the Olympics to a predominantly Muslim country for the first time.
Madrid, bidding for a third straight time, had seemed to have gained the most momentum in recent weeks despite Spain's economic crisis and 27 percent unemployment rate. The Madrid team claimed the games would pose no financial risk because most of the venues were already built.
The Turkish delegation pressed its case of taking the games to a city linking the continents of Europe and Asia.
With the civil war in Syria posing a major issue, Turkish Prime Minister Recep Tayyip Erdogan said selecting Turkey "will send a very meaningful and strong message, not only to the world, but to our broader region."
"At this critical moment, we would like to send a strong message of peace to the whole world from Istanbul," Erdogan said.
Madrid said 80 percent of its venues were ready and only $1.9 billion was needed for construction, a fraction of the other two bids.
"Madrid has perhaps the most reasonable and responsible financial foundation in recent Olympic history," Spanish Prime Minster Mariano Rajoy said. "We can host the Olympics in 2020 with no risk to the Olympic movement."
AP Sports Writers Stephen Wade and Tales Azzoni in Buenos Aires, James Armstrong in Tokyo, Desmond Butler in Istanbul, and Harold Heckle and Joseph Wilson in Madrid contributed to this report.
Follow Stephen Wilson on Twitter: http://twitter.com/stevewilsonap