I found the problem.
I mistakenly registered onvif cameras as private. When I went back to registartion and manually added cams from private to onvif, then the higher resolution choice appeared in Encoding on the NVR.
So now I have all my 8 cams at 3mp and can save the settings.
Problem solved.
When registering Onvif cameras, make sure you register as Onvif, not Private.
Thanks for your help in this NVR IPC mistake on my part.