The link I sent was related to an issue where if you were using multiple processors and one the the processes died no error would be reported, but there would be missing flows. The crash was due to a file issue, so we added some extra output to the command so if a process failed you would be able to see it in the logfile. You could try running the problem file with one processor to see if you have the same issue.
sorry, I missed the link somehow.
I updated to v.1.26 ; with one processor I get 46431 in G7T8VG302cp.scrap.flow and 76254 in G7T8VG302cp.trim.flow
In the logfile there’s an error:
mothur > trim.flows(flow=G7T8VG302cp.flow, oligos=oligosmothur2.tsv, pdiffs=2, bdiffs=1, minflows=360, maxflows=720, processors=1)
Using 1 processors.
10000
20000
…
430000
440000
446202
[ERROR]: Could not open G7T8VG302cp.flow.files
Output File Names:
G7T8VG302cp.trim.flow
G7T8VG302cp.scrap.flow
G7T8VG302cp.Ae.150C.v341F.flow
…
G7T8VG302cp.flow.files
So it looks like it can only process part of the file?
Thanks, Sandra
It looks like its getting to the end of the file, because it outputs 446202. Could you send the flow and oligos file to mothur.bugs@gmail.com so I can try to troubleshoot it for you?
Thanks so much. The flow file is 1.67 GB so I don’t think I can zip it and mail it but I’ve mailed the oligos file.
Is there another way I can send you the flow file?
Thanks, Sandra
I ran your files on my Mac and Windows XP machine with the executable versions of 1.26.0 and was not able to reproduce the problem. I noticed that the portion of the logfile you posted includes: [ERROR]: Could not open G7T8VG302cp.flow.files. Since your other output files opened without error, I assume your permissions are fine for the output folder. Your files are rather large, could you possibly be running out of disk space to write to?
oops that indeed seemed to be the problem, thanks.
I’ve been running shhh.flows for almost two weeks now (!)
shhh.flows(flow=…/data/G7T8VG302cp.flow.files, processors=8)
It’s still running but and looking at one logfile I see this
Finalizing…
Total time to process not found: 20051 20051.7
[ERROR]: Could not open G7T8VG302cp.shhh.fasta4971.num.temp
[ERROR]: main process expected 4971 to complete 4 files, and it only reported completing 0. This will cause file mismatches. The flow files may be too large to process with multiple processors.
[ERROR]: Could not open G7T8VG302cp.shhh.fasta4972.num.temp
[ERROR]: main process expected 4972 to complete 4 files, and it only reported completing 0. This will cause file mismatches. The flow files may be too large to process with multiple processors.
[ERROR]: Could not open G7T8VG302cp.shhh.fasta4973.num.temp
[ERROR]: main process expected 4973 to complete 4 files, and it only reported completing 0. This will cause file mismatches. The flow files may be too large to process with multiple processors.
[ERROR]: Could not open G7T8VG302cp.shhh.fasta4974.num.temp
[ERROR]: main process expected 4974 to complete 4 files, and it only reported completing 0. This will cause file mismatches. The flow files may be too large to process with multiple processors.
[ERROR]: Could not open G7T8VG302cp.shhh.fasta4977.num.temp
[ERROR]: main process expected 4977 to complete 8 files, and it only reported completing 0. This will cause file mismatches. The flow files may be too large to process with multiple processors.
So should I simply repeat with only one processor or could there be other problems?
Thanks!
I would try it with 1 processor. The more processors you have the more memory you need and shhh.flows in rather memory intensive.
thanks, I’m trying that now and it seems to be going fine, albeit slow 