There is currently no method of directly setting the overlap in Topaz extract. One alternative is to use the Inspect particle picks job to try filtering out picks with lower model scores to hopefully reduce the number of overlapping particles.
Other alternatives include retraining the model with different parameters or with training picks that are not as close together.
Regarding what the parameters you mentioned do:
Radius of extracted regions: radius to use for extraction, if a negative value is input, Topaz will calculate a radius using the parameters below
Max assignment radius: maximum distance between prediction and labeled target allowed for considering them a match
Max extraction radius: maximum radius for extraction when calculating which radius to use for extraction
Min extraction radius: minimum radius for extraction when calculating which radius to use for extraction
Step radius: grid size when calculating which radius to use for extraction
Increase the ‘radius of extracted regions’ in ‘Topaz Extract’. This is the radius in pixels after Topaz downsampling. The other parameter you may wish to change is ‘particle threshold’ (the more positive the number, the less objects are picked). You usually shouldn’t need to touch the other parameters.