Tom, the car looks great. For the original paint it appears to be in amazing condition. So you should have no problem buffing it.
Personally I hate the "door moldings" people add to any car, and especially a car like the X. Mine has some that was riveted on (aluminum track with a rubber insert). So I'll have to fill all the holes. In that respect consider yourself lucky.
Considering you will likely have to do some touch up paint, I'd try every solvant I could first (to get the glue off). It can't really hurt any more than sanding, and if you get lucky it might remove the glue without harming the underlying paint.
As for matching the paint. I'd just have it custom matched at a paint shop. After you buff the whole car the color will change, and no matter what you do it won't be the same as it was originally. The odds of finding a correct match by trial and error are very slim. You'll likely end up spending more on a bunch of touch-up bottles.
Just my thoughts. Great car. Those side stripes must be part of the Hugh Hefner special signature Playboy edition?