| | 1 | = Workflow execution / progress data analysis = |
| | 2 | |
| | 3 | == First batch == |
| | 4 | |
| | 5 | === First alignment step === |
| | 6 | |
| | 7 | || '''#''' || '''Sample''' || '''WF''' || '''Status''' || '''Start''' || |
| | 8 | || 1 || first-batch A (17 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-0a2b547d/html/workflow-0a2b547d.html workflow-0a2b547d] || partly done || 29-04-2011 21:40 || |
| | 9 | || 2 || first-batch G (50 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-72df4adf/html/workflow-72df4adf.html workflow-72df4adf] || partly done || 29-04-2011 21:44 || |
| | 10 | || 3 || first-batch (121 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-ecccffa5/html/workflow-ecccffa5.html workflow-ecccffa5] || running || 05-05-2011 17:01 || |
| | 11 | |
| | 12 | === Fastqc === |
| | 13 | |
| | 14 | || '''#''' || '''Sample''' || '''WF''' || '''Status''' || '''Start''' || |
| | 15 | || 1 || first-batch (366 fastq files) || [http://orange.ebioscience.amc.nl/workflows/workflow-4888e67c/html/workflow-4888e67c.html workflow-4888e67c] || running || 09-05-2011 08:57 || |
| | 16 | |
| | 17 | == Second batch == |
| | 18 | |
| | 19 | === First alignment step === |
| | 20 | |
| | 21 | || '''#''' || '''Sample''' || '''WF''' || '''Status''' || '''Start''' || |
| | 22 | || 1 || A4a || [http://orange.ebioscience.amc.nl/workflows/workflow-693426f3/html/workflow-693426f3.html F] [http://orange.ebioscience.amc.nl/workflows/workflow-35a9b777/html/workflow-35a9b777.html F] [http://orange.ebioscience.amc.nl/workflows/workflow-4ba3f651/html/workflow-4ba3f651.html F] [http://orange.ebioscience.amc.nl/workflows/workflow-425d9ceb/html/workflow-425d9ceb.html workflow-425d9ceb] || done || |
| | 23 | || 2 || Vartest || [http://orange.ebioscience.amc.nl/workflows/workflow-490b15f8/html/workflow-490b15f8.html workflow-490b15f8] || done || |
| | 24 | || 3 || Iteration test || [http://orange.ebioscience.amc.nl/workflows/workflow-bf48aff1/html/workflow-bf48aff1.html workflow-bf48aff1] || failed || |
| | 25 | || 4 || Iteration test || [http://orange.ebioscience.amc.nl/workflows/workflow-923c6588/html/workflow-923c6588.html workflow-923c6588] || done || |
| | 26 | || 5 || 60-samples-batch (15 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-d80b5767/html/workflow-d80b5767.html workflow-d80b5767] || 10 / 15 done || 11-02-2011 19:30 || |
| | 27 | || 6 || 60-samples-batch A (55 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-cbaca6e5/html/workflow-cbaca6e5.html workflow-cbaca6e5] || 15 / 55 done || 12-02-2011 13:55 || |
| | 28 | || 7 || 60-samples-batch A remaining 1 (17 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-835250b1/html/workflow-835250b1.html workflow-835250b1] || failed (grid very busy) || 07-03-2011 17:45 || |
| | 29 | || 8 || 60-samples-batch A remaining 1 (17 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-31eb952d/html/workflow-31eb952d.html workflow-31eb952d] || 1/17 done || 08-03-2011 14:15 || |
| | 30 | || 9 || 60-samples-batch A remaining (27 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-fd98c7c8/html/workflow-fd98c7c8.html workflow-fd98c7c8] || 11/27 done || 15-03-2011 10:45 || |
| | 31 | || 10 || 60-samples-batch G (27/54 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-a781209c/html/workflow-a781209c.html workflow-a781209c] || 3/27 done || 15-03-2011 20:37 || |
| | 32 | || 11 || second-batch R10-11-12 (27 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-6fe1cb10/html/workflow-6fe1cb10.html workflow-6fe1cb10] || 6/27 done || 16-03-2011 10:57 || |
| | 33 | || 12 || second-batch R13-14-15 (27 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-77b82882/html/workflow-77b82882.html workflow-77b82882] || 3/27 done || 18-03-2011 19:07 || |
| | 34 | || 13 || second-batch R16-17 (24 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-18ec0160/html/workflow-18ec0160.html workflow-18ec0160] || 7/24 done || 19-03-2011 11:00 || |
| | 35 | || 14 || second-batch R18 (11 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-7596a676/html/workflow-7596a676.html workflow-7596a676] || 2/11 done || 19-03-2011 18:33 || |
| | 36 | || 15 || second-batch R19-20-21 (31 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-58ea16f1/html/workflow-58ea16f1.html workflow-58ea16f1] || 4 done, scheduled downtime, rest will fail || 20-03-2011 11:50 || |
| | 37 | || 16 || second-batch R22-8-9 (altered submission scheme: submits 1 job/5 min) || [http://orange.ebioscience.amc.nl/workflows/workflow-ea6b36a6/html/workflow-ea6b36a6.html workflow-ea6b36a6] || 4 done, scheduled downtime, rest will fail || 20-03-2011 12:37 || |
| | 38 | || 17 || second-batch R10-R17 A9-23 (206 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-7d2b2b5b/html/workflow-7d2b2b5b.html workflow-7d2b2b5b] || 5 done, scheduled downtime, rest will fail || 21-03-2011 11:59 || |
| | 39 | || 18 || second-batch (244 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-c902ebf3/html/workflow-c902ebf3.html workflow-c902ebf3] || This WF is cancelled, because we only want to run these jobs on Gina and HTC || 23-03-2011 09:48 || |
| | 40 | || 19 || second-batch (237 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-62e0f254/html/workflow-62e0f254.html workflow-62e0f254] || 135 done || 25-03-2011 12:20 || |
| | 41 | || 20 || second-batch (102 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-b00af623/html/workflow-b00af623.html workflow-b00af623] || 33 done || 31-03-2011 08:21 || |
| | 42 | || 21 || second-batch (69 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-d5709592/html/workflow-d5709592.html workflow-d5709592] || 4 done || 04-04-2011 18:17 || |
| | 43 | || 22 || second-batch (65 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-7d8743c1/html/workflow-7d8743c1.html workflow-7d8743c1] || 25 done || 06-04-2011 23:24 || |
| | 44 | || 23 || second-batch (40 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-d8cdf017/html/workflow-d8cdf017.html workflow-d8cdf017] || 16 done || 15-04-2011 18:07 || |
| | 45 | || 24 || second-batch (24 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-177917d4/html/workflow-177917d4.html workflow-177917d4] || 12 done || 18-04-2011 11:25 || |
| | 46 | || 25 || second-batch (12 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-289c831f/html/workflow-289c831f.html workflow-289c831f] || 5 done || 20-04-2011 21:15 || |
| | 47 | || 26 || second-batch (7 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-96f414e3/html/workflow-96f414e3.html workflow-96f414e3] || running || 22-04-2011 || |
| | 48 | || 27 || second-batch (3 lanes) || [http://orange.ebioscience.amc.nl/workflows/workflow-63ecec34/html/workflow-63ecec34.html workflow-63ecec34] || running || 27-04-2011 15:22 || |
| | 49 | |
| | 50 | === Fastqc analysis === |
| | 51 | |
| | 52 | || '''#''' || '''Sample''' || '''WF''' || '''Status''' || '''Start''' || |
| | 53 | || 1 || second-batch (2x295 lanes = 590 fastq files) || [http://orange.ebioscience.amc.nl/workflows/workflow-417415cc/html/workflow-417415cc.html workflow-417415cc] || 396 done || 23-04-2011 02:44 || |
| | 54 | || 2 || second-batch (194 fastq files) || [http://orange.ebioscience.amc.nl/workflows/workflow-6ded80d3/html/workflow-6ded80d3.html workflow-6ded80d3] || 189 done || 23-04-2011 11:40 || |
| | 55 | || 3 || second-batch (5 fastq file) || [http://orange.ebioscience.amc.nl/workflows/workflow-2bb76072/html/workflow-2bb76072.html workflow-2bb76072] || 5 done || 23-04-2011 14:40 || |
| | 56 | |
| | 57 | RESULTS: [http://www.bbmriwiki.nl/attachment/wiki/BigCompute/log-fastqc20110423.xls log-fastqc20110423.xls] - contains information about run time and disk usage on the compute nodes and info about the number of sequences per lane |
| | 58 | |
| | 59 | THROUGHPUT: workflow run time was 12 hrs, total CPU run time was 12 days (speedup of ~24x for this component) |
| | 60 | |
| | 61 | STATUS: done |
| | 62 | |
| | 63 | == First and second batch == |
| | 64 | |
| | 65 | === Mark-duplicates analysis on all files that are aligned so far === |
| | 66 | || '''#''' || '''Sample''' || '''WF''' || '''Status''' || '''Start''' || |
| | 67 | || 1 || 351 lanes || [http://orange.ebioscience.amc.nl/workflows/workflow-a5a7a078/html/workflow-a5a7a078.html workflow-a5a7a078] || 212 done || 04-05-2011 12:54 || |
| | 68 | || 2 || 139 lanes || [http://orange.ebioscience.amc.nl/workflows/workflow-87b3994d/html/workflow-87b3994d.html workflow-87b3994d] || 115 done || 05-05-2011 15:29 || |
| | 69 | || 3 || 24 lanes || [http://orange.ebioscience.amc.nl/workflows/workflow-7e395331/html/workflow-7e395331.html workflow-7e395331] || failed || 06-05-2011 08:22 || |
| | 70 | || 4 || 24 lanes || [http://orange.ebioscience.amc.nl/workflows/workflow-6207c67d/html/workflow-6207c67d.html workflow-6207c67d] || 19 done || 06-05-2011 12:55 || |
| | 71 | || 5 || 5 lanes || [http://orange.ebioscience.amc.nl/workflows/workflow-62e2653e/html/workflow-62e2653e.html workflow-62e2653e] || running || 07-05-2011 14:08 || |
| | 72 | |
| | 73 | == Notes == |
| | 74 | |
| | 75 | '''Monitor clusters''' |
| | 76 | * [http://ganglia.sara.nl/?m=load_one&r=week&s=descending&c=LifeScience+Grid&h=&sh=1&hc=4&z=small Ganglia - LifeScience grid] |
| | 77 | * [http://ganglia.sara.nl/?m=load_one&r=week&s=descending&c=GINA+Cluster&h=&sh=1&hc=4&z=small Ganglia - Gina cluster] |
