It turns out the the sweetness of crab meat and the sweetness of a good olive oil clash. And the bitterness of the the olive oil.... even with the best, the bitterness somehow swims to the top.
Sort of like the Uni + Beer problem. If you have had uni and beer, you will say you don't like uni. They bring out the worst in each other. The sushi bar drink of choice with uni is nothing or Sake... but NEVER EVER beer.