The number reverting to the default could be due a bug with setting a value using the arrows. At least in Firefox, if you set any value with the arrows, it is not saved unless you after setting it first click inside the field (to place the cursor), and then click somewhere else.
Did you set the value to 2 or 5 using the arrows, @Inko ? If so, try writing the number instead and see if it works.
I haven’t seen the problem for exactly this setting, but it’s been discussed for other settings before and I think it’s general: 4.4.1 Only single GPU out of 4 is available to select in "run on specific GPU" in Queue - #10 by boggild