Yes - as others have said - shaving the cam box and/or head WILL result in the cam timing being slightly retarded. This is generally not a good thing. If anything, on these engines you would want the cam timing to be slightly advanced for better performance. General consensus is that about 2 degrees advanced is ideal. Zero is just fine. Anything retarded is bad.
Having said that - you probably would not notice a couple of degrees retarded outside of a dyno test. And thus probably not worth the hassle and expense (by using an adjustable cam pulley) to correct it. At least for the small amount of cam box milling you would be likely to do.
If....you are sure you can obtain correct valve clearance by using shims in the 4.9 to 4.9mm range, you will probably be okay. Shims are available up to 5.05mm. Yes you at right near the top of available shims, but probably okay. In the future, any wear will most likely result in the valves recessing a bit and thus requiring thinner shims for any future valve adjustments. Yeah, the thicker shims are a tad heavier, but unless you are racing or using constant very high RPMs , I dont think it is worth the hassle and expense to shave the cam box to allow thinner shims. Which would also then alter your cam timing for the worse.
However..having said that...I would highly advise you to check two other things first !! The stock gasket between the head and cam box is just a thin paper gasket. It works just fine....BUT....only if both the mating surfaces are PERFECTLY flat. You said earlier the bottom of you head was warped and you had it machined straight and flat. That is good...but..often in this case the TOP of the head - where the cam box bolts onto - is also warped and no longer perfectly flat !!! Your machine shop should have checked this, but often they dont. And/or your cam box may be warped too. If either the top of the head or the bottom of your cam box are not perfectly flat, then that thin paper gasket will NOT seal that joint perfectly, and you will leak oil from there. Which can be very annoying.....
So..if and only if...you need to have the head and/or cam box machined to make them flat again..do it. This will also bring your required shims down to a better range. Other than that.....dont bother.....