Police have said the man’s death appeared to be a “tragic accident”.
Emergency services were called at about 8.30am.
One witness reported seeing the police arrive and seeing blue lights.
He said a tent was set up, and officers appeared to be trying to recover a body from the water.
Paul Budd, 63, who lives in Brighton Marina, said: “There were two ambulances and a paramedic car on the ramp leading up to the East Jetty.
“Then two police cars arrived and, as I was walking away to get a bus, the lifeboat people turned up as well.
“I was wondering what was going on and it would make sense that a body had been pulled from the water.
“I knew something was wrong because we never get police down this way.”
A Coastguard team from Shoreham joined a volunteer crew from Brighton RNLI, the police and the ambulance service.
A police spokesman said: “Emergency services attended and the man was sadly pronounced dead at the scene.
“His body was recovered and the coroner’s officer has been informed.
“There are no suspicious circumstances and early indications suggest it was a tragic accident.”
A Coastguard spokesman said: “At 8.40am today the ambulance service asked for our assistance at Brighton Marina.
“Shoreham Coastguard Rescue Team, Brighton RNLI Lifeboat were sent to the scene, alongside Sussex Police and the South East Ambulance Service.”