I like to reuse my trained Topaz model to pick on new datasets by importing my previous job into a new project. By default this writes the preprocessed micrographs to the original Topaz Train job from the project that I exported from. Would it be possible to include an option to write the Topaz preprocess micrographs to the Topaz Extract job instead of the Topaz Train job? Or perhaps there is another solution that I am not aware of.
Thanks!
Evan
@emgreen During job export, were all linked files “dereferenced”, in the sense of the -h/--dereference option of tar, as discussed in the tutorial?
Hi @wtempel, thanks for the help. For the specific test case I was investigating I did not “dereference” my exported job, but the original data was still live on my storage space. I think I wasn’t totally clear in my original post, here is what I noticed in more detail.
The original topaz job was in:
/fsx/old/P33/J86
The cryosparc output directory for my current project is:
#SBATCH --output=/fsx/new/P45/J6/job.log
But Topaz preprocess does not write to my new project directory:
Starting micrograph preprocessing by running command /home/ec2-user/anaconda3/envs/topaz/bin/topaz preprocess --scale 4 --niters 200 --num-workers 8 -o /fsx/old/P33/J86/preprocessed
Topaz extract does write to the correct directory though:
Starting extraction by running command /home/ec2-user/anaconda3/envs/topaz/bin/topaz extract --radius 16 --threshold -6 --up-scale 4 --assignment-radius -1 --min-radius 5 --max-radius 100 --step-radius 5 --num-workers 8 --device 0 --model /fsx/new/P45/import/P33_J86_topaz_train/P33_J86_topaz_model/J86/models/model_epoch10.sav -o /fsx/new/P45/J6/topaz_particles_prediction.txt
I hope that is more clear. Based on these results, I’m not sure if “dereferencing” would make a difference.
@emgreen Thank you for bringing this issue to our attention. This is indeed a bug, and we have taken note.