Export project metadata to csv

Features and Functionality
1

Hi,

As mentioned in this post (Job dependencies were lost after importing projects), the parent/child information for jobs were lost when importing a project. Not sure if it just happens when importing into an old (<v3.2) database backup, as I have done. I have more than a thousand jobs to go through and if the parent/child links cannot be restored somehow, it would really help if I can export the metadata for all jobs to a list as suggested here by @olibclarke UI bugs with tree view of cryoSPARC v2). From this old post by @sdawood Note taking capability for the project) it seems like a csv export of metadata was on the cards. Any update on this?

image
image1004×211 21.8 KB

2

Hi @lizellelubbe,

Thanks for your post and sorry for the trouble. We’re looking into the root cause of the original issue you posted, but I can provide an update on the metadata export feature. CSV output functionality has been built (and works quite well!). It allows you to download the full list of whatever projects, workspaces or jobs you are browsing. It, along with many other enhancements to the user interface are being tested internally by our team and will be released as soon as possible. I can’t provide a concrete date at this time.

In the meantime, you can access the same information by querying the cryoSPARC database (MongoDB). The mongoexport command-line program allows for outputting a JSON file with all matching results:

./cryosparc2_package/cryosparc_master/deps/external/mongodb/bin/mongoexport \
  --port=40001 \
  --db=meteor \
  --collection=jobs \
  --query='{ "project_uid": "P1" }' \
  --fields='uid,job_type,status,parents,children' \
  --out=jobs.json

The example above outputs a JSON file containing all jobs within a particular project.

- Suhail

1 Like
3

Hi @sdawood,
Thanks for the reply and for looking into the original parent/child issue. Great news about the upcoming metadata export feature! I just ran mongoexport and it worked really well - glad to now have a list of my job descriptions - good thing they were detailed. It helps a lot :slight_smile:

1 Like
4

Sorry @sdawood but is any further editing required after creating the JSON file? It’s not a cryoSPARC issue but thought you might have a tip. The JSON file cannot import to Excel and complains about a syntax error when read by a JSON parser (SyntaxError: Unexpected token { in JSON at position 189)

5

Hi @lizellelubbe,

It may have something to do with the special JSON syntax MongoDB uses. I realized that the mongoexport tool can also output CSV files. You can modify the script by adding this parameter --type=csv and editing the output filename to .csv instead of .json. Hopefully that works!

- Suhail

1 Like
6

Yes thank you that worked!