MPI, multi-cores, oh my ...

The SOP page is fantastic - thank you for putting it together! However, now that you’ve convinced me that shhh.flows would be good to use (or at least try out), how do I implement it?

One page says not to use shhh.flows unless you have multi-cores or mpi. Are dual cores sufficient? Or are we talking more along the lines of a computing cluster? And the open-mpi page tells the truly ignorant very little about what it does and what situations it is useful in. In addition, I’ve seen a couple of places where people have said not to use mpi if you’re running mothur on a single computer. Uhuh …

I would be extremely grateful if someone would spell this all out for me. When is it appropriate to use mpi? When is it not? And if mpi is not appropriate for the average dual core pc user, can we still use shhh.flows? What kind of time lag would I be looking at? I’m used to running the full Costello stool analysis on a plate’s worth of data in 3-4 hours, but I would be ok with running overnight if I knew what to expect. Tying up my pc for days at a time = less acceptable.

And as always, thank you!

On the SOP page I found the following: It took me about 8 hours (wall time) to run shhh.flows on the 96 samples in GQY1XT001.flow.files with 8 processors.

So could I take this to mean that a rough estimate of time to run shhh.flows on your average dual core pc would be something like 32 hours? I’m sorry to keep worrying at this, but a) it’s frustrating not to be able to find this completely explained in plain language in one location and b) I’m a little concerned by the idea that it appears the SOP is including a step that may only be available to people with access to a cluster. I love mothur because it’s so very accessible to everyone. I think it’s fantastic that it now includes options that utilize more powerful options like clusters, but to include it in the SOP as stands seems to deviate from that “Power to the People” spirit somewhat.

Or maybe I’m romanticizing a piece of software a little too much.

The first option should be a lot faster than the second because it is fully parallelized, but in reality it really doesn’t seem to matter much. It took me about 8 hours (wall time) to run shhh.flows on the 96 samples in GQY1XT001.flow.files with 8 processors.

So you can use mpi, but it doesn’t really help a ton. If you have a dual core processor (i.e. your laptop) you can go for it, but yeah, it may take a while. Probably not exactly 32 hours, but it will slow down your World of Warcraft play :). In our lab, we don’t use mpi and just set processors to however many processors that computer has. Part of the reason I left the trim.seqs approach in the SOP as an alternative to shhh.flows was for situations like this; however, I am conflicted because in the grand scheme of things 32 hrs isn’t that long, and you get a lot more reads and they’re longer using shhh.flows.

Hope this helps,