Since your accessories are neutral, I think you'd be all right going with a darker, bold color. However, since yellow is such a bright color in itself, you probably don't need to emphasize that by going with a shade that's too striking, like black or red.
I'm thinking a strong brown, like russet or chocolate, would match well. Also, call me crazy, but would a medium shade of green (such as forest green) go with it?