Well, I managed to solve this problem. Not sure exactly what it was. I tried bending the pins in the amp connector slots to ensure a good contact, had no luck, took the amp apart, looked OK, put it back together, bent the pins more then reinstalled and it has worked perfect ever since.
BTW, there is not much inside that factory amp... the weight of it is all the steel case.
Also, there has been talk on the board about how to install aftermarket rear deck speakers with the infinity system. You just have to do a bit of semi-custom wiring. Get a two way speaker and solder the wires originally from the small, top speaker to the new little tweeter, and the wires from the big speaker to the new full 6x9 speaker. It has been mentioned that you can do this, but it seemed that there was also some controversy from the old posts I dug up. For the record, it sounds great.