Took your suggestion and had lunch at the Fish Market today. I ordered the Sea Bass with steamed vegetables and Jasmine rice. Also had a cup of their famous white clam chowder. Their clam chowder is definitely the best. I'll take a picture of my order next time to share.
Stay away from the shark, they're not considered a sustainable seafood source because they're very slow reproducers, and also can be high in mercury. A great alternative is wild Alaskan salmon or farm-raised tilapia. If you go for lobster, try to find local lobster to support local San Diego fishermen! Pt. Loma Seafood is a great place to find local seafood.
Blue Water Seafood on India street has really great seafood sandwiches and salads. Can't see the waterfront from there, but it's close to the Port Admin building and prices are reasonable
My vote is for the Fish Market! Everything on the menu is superfresh, The clam chowder is the best in San Diego (with Omni San Diego a close second but not on the waterfront) and a delicious and reasonably priced lunch is the chowder with rolls and butter. YUM!!!