The vessel master told investigators the yacht was rolled several times by the waves, the Coast Guard said.
A Mayday call went out at about 3 p.m. PDT on Saturday.
The entire crew was believed to have been wearing life vests and foul weather gear, which made rescuers optimistic they might find survivors.
The Farallon Islands are a series of steep, rocky outcroppings visible from San Francisco on a clear day and uninhabited except for a manned research station. Part of a national wildlife refuge and closed to the public, the islands are home to vast quantities of sea birds and are surrounded by waters rich with seals and sea lions, and sharks that feed on them.
Search crews have encountered plenty of wildlife in their search for the missing crew members, but have not reported seeing any sharks that would pose additional danger to anyone stuck in the water, the Coast Guard said.
Low Speed Chase remains grounded on one of the islands but reportedly intact, the guard said.
R. David Britt, a University of California, Davis chemist who skippered his sailboat, Split Water, in the Full Crew Farallones Race for the third time on Saturday, described the sailing out by the islands that day as "pretty intense." Swells nearing 20-feet-high were breaking far enough from the craggy outcroppings that Britt says he steered farther around them than he otherwise might to avoid getting swamped by a wave or dashed onto the rocks.
"The worst thing is to have a wave break on you," he said. "You can go up and down, up and down, but if a wave breaks on the cockpit on top of the crew, that's how somebody could get swept out of the boat."
Britt thinks he was not far ahead of Low Speed Chase as they rounded the islands, and thought it strange when he looked back later and no longer saw his competitor.
During the day, people dropped roses and tulips by the entrance of the San Francisco Yacht Club, which hosted a members-only candlelight vigil and prayer service Sunday evening to honor the missing crew members and the one who died.
Anne Kasanin, the mother of the sailor who died, attended the service and was touched by how many people knew her son, who started sailing at age 7 and lived his whole life on the cove where the yacht club is located. He was a well-known local artist whose landscapes in acrylic and oil reflected his love of the water, she said.
"He was a very dear son to me and a tremendous help, and I'm going to miss him very much," she said.
Bradford and the other two survivors attended the vigil, but were too distraught to talk about their experience, Lynch said.
Club member Brian Swift said that even though sailors are aware of the dangers of racing in open ocean, "what everybody is feeling is utter shock."
The San Francisco Bay area is home to a vibrant sailing scene, with more than 35 yacht clubs ringing the bay's chilly, wind-whipped waters. Due to steady winds, easy access and a picturesque backdrop featuring the Golden Gate Bridge, Alcatraz and Coit Tower, the city of San Francisco was chosen to host the 2013 America's Cup, the sport's marquee event.
There are dangers, however, such as strong tides and commercial shipping. Those dangers, including strong winds, increase when sailors are on the open ocean beyond the Golden Gate Bridge.
Results from last year's Full Crew Farallones Race posted on the web site of the Yacht Racing Association of San Francisco Bay shows that Bradford entered Low Speed Chase in the event, but did not finish it.
Marshall reported from Belvedere, Calif., Bernie Wilson in San Diego contributed to this report.
Copyright 2012 The Associated Press. All rights reserved. This material may not be published, broadcast, rewritten or redistributed.