Machine out the speaker pocket with a milling machine or other method of metal removal.
My secret to milling model train frames is to use an aluminum roughing bit. The frames are made from a "sticky" metal, typically Zamac, that doesn't like to chip. The roughing bit cuts them like butter.
A spritz of WD-40 is about all I ever need for cutting lubricant and I do that sparingly.
I also don't get too hung up on absolute precision. Just cut the frame... 0.1 ounces of extra metal will not translate to enough traction to pull a dozen extra cars. I do try to be neat and tidy though.