Dear @Yuqi,
These artefacts commonly occur when the mask hugs the protein too closely and the width of the padded edge is too small for the dataset resolution. You can try manually making a mask using the Volume Tools job from the Ab-initio volume with a very soft falloff by increasing the pad width. The dilation width is less relevant; the padding is what you want to increase.
A good measure of how successful refinement is likely to be is whether you see secondary structure in your trials of Ab-initio. Other than that, the standard culprits apply to the loss of detail between 2D classification and ab-initio (discrete and/or continuous heterogeneity, poor orientation distribution, etc.).
Best, Michael