= Workflow execution / progress data analysis =

== First batch ==

=== First alignment step ===

|| '''#''' || '''Sample''' || '''WF''' || '''Status''' || '''Start''' ||
|| 1 || first-batch A (17 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-0a2b547d/html/workflow-0a2b547d.html workflow-0a2b547d] || partly done || 29-04-2011 21:40 ||
|| 2 || first-batch G (50 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-72df4adf/html/workflow-72df4adf.html workflow-72df4adf] || partly done || 29-04-2011 21:44 ||
|| 3 || first-batch (121 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-ecccffa5/html/workflow-ecccffa5.html workflow-ecccffa5] || 28 done || 05-05-2011 17:01 ||
|| 4 || first-batch (93 lanes + 1 from second batch) || [http://orange.ebioscience.amc.nl/workflows/workflow-9c7ae662/html/workflow-9c7ae662.html workflow-9c7ae662] || running || 09-05-2011 09:50 ||

=== Fastqc ===

|| '''#''' || '''Sample''' || '''WF''' || '''Status''' || '''Start''' ||
|| 1 || first-batch (366 fastq files) || [http://orange.ebioscience.amc.nl/workflows/workflow-4888e67c/html/workflow-4888e67c.html workflow-4888e67c] || 140 done || 09-05-2011 08:57 ||
|| 2 || first-batch (226 fastq files) || [http://orange.ebioscience.amc.nl/workflows/workflow-b8a55600/html/workflow-b8a55600.html workflow-b8a55600] || running || 11-05-2011 13:05 ||

== Second batch ==

=== First alignment step ===

|| '''#''' || '''Sample''' || '''WF''' || '''Status''' || '''Start''' ||
|| 1 || A4a || [http://orange.ebioscience.amc.nl/workflows/workflow-693426f3/html/workflow-693426f3.html F] [http://orange.ebioscience.amc.nl/workflows/workflow-35a9b777/html/workflow-35a9b777.html F] [http://orange.ebioscience.amc.nl/workflows/workflow-4ba3f651/html/workflow-4ba3f651.html F] [http://orange.ebioscience.amc.nl/workflows/workflow-425d9ceb/html/workflow-425d9ceb.html workflow-425d9ceb] || done ||
|| 2 || Vartest || [http://orange.ebioscience.amc.nl/workflows/workflow-490b15f8/html/workflow-490b15f8.html workflow-490b15f8] || done ||
|| 3 || Iteration test || [http://orange.ebioscience.amc.nl/workflows/workflow-bf48aff1/html/workflow-bf48aff1.html workflow-bf48aff1] || failed ||
|| 4 || Iteration test || [http://orange.ebioscience.amc.nl/workflows/workflow-923c6588/html/workflow-923c6588.html workflow-923c6588] || done ||
|| 5 || 60-samples-batch (15 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-d80b5767/html/workflow-d80b5767.html workflow-d80b5767] || 10 / 15 done || 11-02-2011 19:30 ||
|| 6 || 60-samples-batch A (55 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-cbaca6e5/html/workflow-cbaca6e5.html workflow-cbaca6e5] || 15 / 55 done || 12-02-2011 13:55 ||
|| 7 || 60-samples-batch A remaining 1 (17 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-835250b1/html/workflow-835250b1.html workflow-835250b1] || failed (grid very busy) || 07-03-2011 17:45 ||
|| 8 || 60-samples-batch A remaining 1 (17 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-31eb952d/html/workflow-31eb952d.html workflow-31eb952d] || 1/17 done || 08-03-2011 14:15 ||
|| 9 || 60-samples-batch A remaining (27 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-fd98c7c8/html/workflow-fd98c7c8.html workflow-fd98c7c8] || 11/27 done || 15-03-2011 10:45 ||
|| 10 || 60-samples-batch G (27/54 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-a781209c/html/workflow-a781209c.html workflow-a781209c] || 3/27 done || 15-03-2011 20:37 ||
|| 11 || second-batch R10-11-12 (27 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-6fe1cb10/html/workflow-6fe1cb10.html workflow-6fe1cb10] || 6/27 done || 16-03-2011 10:57 ||
|| 12 || second-batch R13-14-15 (27 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-77b82882/html/workflow-77b82882.html workflow-77b82882] || 3/27 done || 18-03-2011 19:07 ||
|| 13 || second-batch R16-17 (24 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-18ec0160/html/workflow-18ec0160.html workflow-18ec0160] || 7/24 done || 19-03-2011 11:00 ||
|| 14 || second-batch R18 (11 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-7596a676/html/workflow-7596a676.html workflow-7596a676] || 2/11 done || 19-03-2011 18:33 ||
|| 15 || second-batch R19-20-21 (31 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-58ea16f1/html/workflow-58ea16f1.html workflow-58ea16f1] || 4 done, scheduled downtime, rest will fail || 20-03-2011 11:50 ||
|| 16 || second-batch R22-8-9 (altered submission scheme: submits 1 job/5 min) || [http://orange.ebioscience.amc.nl/workflows/workflow-ea6b36a6/html/workflow-ea6b36a6.html workflow-ea6b36a6] || 4 done, scheduled downtime, rest will fail || 20-03-2011 12:37 ||
|| 17 || second-batch R10-R17 A9-23 (206 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-7d2b2b5b/html/workflow-7d2b2b5b.html workflow-7d2b2b5b] || 5 done, scheduled downtime, rest will fail || 21-03-2011 11:59 ||
|| 18 || second-batch (244 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-c902ebf3/html/workflow-c902ebf3.html workflow-c902ebf3] || This WF is cancelled, because we only want to run these jobs on Gina and HTC || 23-03-2011 09:48 ||
|| 19 || second-batch (237 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-62e0f254/html/workflow-62e0f254.html workflow-62e0f254] || 135 done || 25-03-2011 12:20 ||
|| 20 || second-batch (102 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-b00af623/html/workflow-b00af623.html workflow-b00af623] || 33 done || 31-03-2011 08:21 ||
|| 21 || second-batch (69 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-d5709592/html/workflow-d5709592.html workflow-d5709592] || 4 done || 04-04-2011 18:17 ||
|| 22 || second-batch (65 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-7d8743c1/html/workflow-7d8743c1.html workflow-7d8743c1] || 25 done || 06-04-2011 23:24 ||
|| 23 || second-batch (40 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-d8cdf017/html/workflow-d8cdf017.html workflow-d8cdf017] || 16 done || 15-04-2011 18:07 ||
|| 24 || second-batch (24 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-177917d4/html/workflow-177917d4.html workflow-177917d4] || 12 done || 18-04-2011 11:25 ||
|| 25 || second-batch (12 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-289c831f/html/workflow-289c831f.html workflow-289c831f] || 5 done || 20-04-2011 21:15 ||
|| 26 || second-batch (7 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-96f414e3/html/workflow-96f414e3.html workflow-96f414e3] || 4 done || 22-04-2011 ||
|| 27 || second-batch (3 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-63ecec34/html/workflow-63ecec34.html workflow-63ecec34] || 1 done || 27-04-2011 15:22 ||
|| 28 || second-batch (2 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-dd84f331/html/workflow-dd84f331.html workflow-dd84f331] || 1 done || 02-05-2011 11:05 ||
|| 29 || second-batch (1 lane) || [http://orange.ebioscience.amc.nl/workflows/workflow-8a720670/html/workflow-8a720670.html workflow-8a720670] || running || 05-05-2011 16:58 ||

=== Fastqc analysis ===

|| '''#''' || '''Sample''' || '''WF''' || '''Status''' || '''Start''' ||
|| 1 || second-batch (2x295 lanes = 590 fastq files) || [http://orange.ebioscience.amc.nl/workflows/workflow-417415cc/html/workflow-417415cc.html workflow-417415cc] || 396 done || 23-04-2011 02:44 ||
|| 2 || second-batch (194 fastq files) || [http://orange.ebioscience.amc.nl/workflows/workflow-6ded80d3/html/workflow-6ded80d3.html workflow-6ded80d3] || 189 done || 23-04-2011 11:40 ||
|| 3 || second-batch (5 fastq file) || [http://orange.ebioscience.amc.nl/workflows/workflow-2bb76072/html/workflow-2bb76072.html workflow-2bb76072] || 5 done || 23-04-2011 14:40 ||

RESULTS: [http://www.bbmriwiki.nl/attachment/wiki/BigCompute/log-fastqc20110423.xls log-fastqc20110423.xls] - contains information about run time and disk usage on the compute nodes and info about the number of sequences per lane

THROUGHPUT: workflow run time was 12 hrs, total CPU run time was 12 days

STATUS: done

== First and second batch ==

=== Mark-duplicates analysis on all files that are aligned so far ===
|| '''#''' || '''Sample''' || '''WF''' || '''Status''' || '''Start''' ||
|| 1 || 351 lanes || [http://orange.ebioscience.amc.nl/workflows/workflow-a5a7a078/html/workflow-a5a7a078.html workflow-a5a7a078] || 212 done || 04-05-2011 12:54 ||
|| 2 || 139 lanes || [http://orange.ebioscience.amc.nl/workflows/workflow-87b3994d/html/workflow-87b3994d.html workflow-87b3994d] || 115 done || 05-05-2011 15:29 ||
|| 3 || 24 lanes || [http://orange.ebioscience.amc.nl/workflows/workflow-7e395331/html/workflow-7e395331.html workflow-7e395331] || failed || 06-05-2011 08:22 ||
|| 4 || 24 lanes || [http://orange.ebioscience.amc.nl/workflows/workflow-6207c67d/html/workflow-6207c67d.html workflow-6207c67d] || 19 done || 06-05-2011 12:55 ||
|| 5 || 5 lanes || [http://orange.ebioscience.amc.nl/workflows/workflow-62e2653e/html/workflow-62e2653e.html workflow-62e2653e] || failed || 07-05-2011 14:08 ||
|| 6 || 37 lanes || [http://orange.ebioscience.amc.nl/workflows/workflow-05988a60/html/workflow-05988a60.html workflow-05988a60] || running || 11-05-2011 15:00 ||

=== Samtools flagstat on bam files ===

|| '''#''' || '''Sample''' || '''WF''' || '''Status''' || '''Start''' ||
|| 1 || 381 lanes || [http://orange.ebioscience.amc.nl/workflows/workflow-a7a0acbf/html/workflow-a7a0acbf.html workflow-a7a0acbf] || 181 done || 09-05-2011 16:56 ||
|| 2 || 200 lanes || [http://orange.ebioscience.amc.nl/workflows/workflow-39699118/html/workflow-39699118.html workflow-39699118] || running || 11-05-2011 14:46 ||


== Notes ==

'''Monitor clusters'''
* [http://ganglia.sara.nl/?m=load_one&r=week&s=descending&c=LifeScience+Grid&h=&sh=1&hc=4&z=small Ganglia - LifeScience grid]
* [http://ganglia.sara.nl/?m=load_one&r=week&s=descending&c=GINA+Cluster&h=&sh=1&hc=4&z=small Ganglia - Gina cluster]