Webapp status reverts from "running" to "starting"

Troubleshooting
1

Over the weekend cryoSPARC stopped responding to web requests. On restart, everything appears fine at first, but web requests get no response. Right after starting, webapp status is RUNNING but shortly after reverts to STARTING. Thereafter seems to roughly alternate between these conditions (must be retrying).

Has anyone else seen this?

My meteor.stdout contains the following error, in the middle of the “generating datasets” stage. It turns out it always fails after the same datasetId. How can I track down that dataset based on the Id?

<=== clipped ===>
datasetId: Kpvcn3YEyS8qPwHuq

/mnt/cache/cryosparc/cryosparc/cryosparc-webapp/bundle/programs/server/node_modules/fibers/future.js:313
                                            throw(ex);
                                            ^
MongoError: Executor error during find command: OperationFailed: Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit.
at Object.Future.wait (/mnt/cache/cryosparc/cryosparc/cryosparc-webapp/bundle/programs/server/node_modules/fibers/future.js:449:15)
at SynchronousCursor._nextObject (packages/mongo/mongo_driver.js:1024:47)
at SynchronousCursor.forEach (packages/mongo/mongo_driver.js:1058:22)
at SynchronousCursor.map (packages/mongo/mongo_driver.js:1068:10)
at SynchronousCursor.fetch (packages/mongo/mongo_driver.js:1092:17)
at Cursor.(anonymous function) (packages/mongo/mongo_driver.js:907:44)
at Cursor.MeteorCursor.(anonymous function) [as fetch] (packages/peerlibrary_reactive-mongo.js:105:27)
at _updateExperiments (server/modules/update-collections.js:107:69)
at Object.updateCollections (server/modules/update-collections.js:4:3)
at _updateCollections (server/modules/startup.js:46:47)
- - - - -
at Function.MongoError.create (/mnt/cache/cryosparc/cryosparc/cryosparc-webapp/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/error.js:31:11)
at queryCallback (/mnt/cache/cryosparc/cryosparc/cryosparc-webapp/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/cursor.js:212:36)
at /mnt/cache/cryosparc/cryosparc/cryosparc-webapp/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/pool.js:455:18
at nextTickCallbackWith0Args (node.js:420:9)
at process._tickCallback (node.js:349:13)
2

Hey Daniel,

Can you try the following:

eval $(cryosparc env)
cd /path/to/cryosparc
source config.sh
mongo localhost:$CRYOSPARC_MONGO_PORT

then you’ll be in the mongoDB shell. There, use the commands:

use meteor
db.experiments.createIndex({'createdAt':1})
exit

this should create the required index and then stopping and starting cryosparc should be okay.
Let us know!
Ali

1 Like
3

We’re up and running again. Thanks!

4

I have the same issue, but after trying what Ali suggested, I got this error message:

Any idea?

Thanks,

Marco

MongoDB shell version: 3.2.9
connecting to: localhost:38001/test
2017-09-17T07:39:17.839-0700 W NETWORK [thread1] Failed to connect to 127.0.0.1:38001, reason: errno:111 Connection refused
2017-09-17T07:39:17.839-0700 E QUERY [thread1] Error: couldn’t connect to server localhost:38001, connection attempt failed :
connect@src/mongo/shell/mongo.js:231:14
@(connect):1:6

exception: connect failed