{is} prepare for rsem docs (#709)

Documentation for prepare_for_em (Was prepare-for-rsem)

* Renamed prepare-for-rsem -> prepare_for_em.
* Added docstring documentation.
* Rearraged common option groups so that CRAM options can be added to prepare_for_em without adding other common sam/bam options. 
* Added docs/reference/prepare_for_em.rst
* Altered index.rst, README and umi_tools.py to recognise the new command
* Updated tests/*_help to account for changed option groups.

Author: IanSudbery

README.rst

@@ -27,7 +27,7 @@ fastq containg UMIs +/- cell barcodes for alignment.
       barcode, for example a library barcode, is left on the read. Can
       also filter reads by quality or against a whitelist (see above)
 
-The remaining commands, ``group``, ``dedup`` and ``count``/``count_tab``, are used to
+The commands, ``group``, ``dedup`` and ``count``/``count_tab``, are used to
 identify PCR duplicates using the UMIs and perform different levels of
 analysis depending on the needs of the user. A number of different UMI
 deduplication schemes are enabled - The recommended method is
@@ -50,6 +50,10 @@ deduplication schemes are enabled - The recommended method is
 * count_tab:
    **As per count except input is a flatfile**
 
+Finally, the command ``prepare_for_em`` will take the output of ``group`` or 
+``dedup`` (after it has been name sorted or collated) and make the output
+compatible with EM tools like Salmon or RSEM. 
+
 See `QUICK_START.md <./doc/QUICK_START.md>`_ for a quick tutorial on
 the most common usage pattern.
 

doc/index.rst

@@ -61,7 +61,7 @@ for alignment.
       barcode, for example a library barcode, is left on the read. Can
       also filter reads by quality or against a whitelist (see above)
 
-The remaining commands, ``group``, ``dedup`` and
+The commands, ``group``, ``dedup`` and
 ``count``/``count_tab``, are used to identify PCR duplicates using the
 UMIs and perform different levels of analysis depending on the needs
 of the user. A number of different UMI deduplication schemes are
@@ -87,6 +87,10 @@ deduplication schemes see :doc:`the_methods`
 * count_tab:
    **As per count except input is a flatfile**
 
+Finally the command ``prepare_for_em`` takes the output of dedup or group and ensures
+that the order of reads and the matching of pairs is compatible with EM algorithms
+such as RSEM and Salmon. 
+
 .. toctree::
    :maxdepth: 1
    :caption: Commands:
@@ -97,6 +101,7 @@ deduplication schemes see :doc:`the_methods`
    dedup <reference/dedup>
    count <reference/count>
    count_tab <reference/count_tab>
+   prepare_for_em <reference/prepare_for_em>
 
 Each tool has a set of :doc:`common_options` for input/output,
 profiling and debugging.
@@ -140,7 +145,7 @@ Enter repository and run:
 
 .. code:: bash
 
-   $ python setup.py install
+   $ pip install .
 
 For more detail see :doc:`INSTALL`
 

doc/reference/prepare_for_em.rst

@@ -0,0 +1,4 @@
+Each tool has a set of :ref:`common_options` for input/output,
+profiling and debugging.
+
+.. automodule:: umi_tools.prepare_for_em

tests/count_help

@@ -67,29 +67,25 @@ Options:
                         File mapping transcripts to genes (tab separated)
     --per-cell          group/dedup/count per cell
 
-  SAM/BAM options:
+  Input/Output format options:
     --in-format=IN_FORMAT
                         File format of the input file. Format is usually
                         implied from the extension of the filename, but maybe
                         overridden with this option. Default=bam
-    --out-format=OUT_FORMAT
-                        File format of the input file. Format is usually
-                        implied from the extension of the filename, but maybe
-                        overridden with this option. Default=bam
     --input-options=INPUT_OPTIONS
                         Format string provided to htslib for reading. Mostly
                         useful for CRAM formatted files. See samtools
                         documentation
-    --output-options=OUTPUT_OPTIONS
-                        Format string provided to htslib for writing. Mostly
-                        useful for CRAM formatted files. See samtools
-                        documentation
+    -i, --in-sam        [Deprecated] Input file is in sam format
+                        [default=False]
     --reference-filename=REFERENCE_FILENAME
                         File path or URL to the genome reference to be used
                         when reading or writing CRAM files. By default, when
                         reading a CRAM file, the reference recorded in the
                         input file will be used unless this is specified. When
                         writing, specifying a reference location is required.
+
+  SAM/BAM filtering options:
     --mapping-quality=MAPPING_QUALITY
                         Minimum mapping quality for a read to be retained
                         [default=0]
@@ -98,9 +94,7 @@ Options:
                         read1, whether or not the template length is the same
     --chrom=CHROM       Restrict to one chromosome
     --subset=SUBSET     Use only a fraction of reads, specified by subset
-    -i, --in-sam        Input file is in sam format [default=False]
     --paired            paired input BAM. [default=False]
-    -o, --out-sam       Output alignments in sam format [default=False]
     --no-sort-output    Don't Sort the output
 
   Dedup and Count SAM/BAM options:
@@ -114,7 +108,7 @@ Options:
                         How to handle unpaired reads. Options are 'discard'or
                         'use' [default=use]
 
-  input/output options:
+  Input/Output pipe options:
     -I FILE, --stdin=FILE
                         file to read stdin from [default = stdin].
     -L FILE, --log=FILE

tests/count_tab_help

@@ -34,7 +34,25 @@ Options:
     --read-length       use read length in addition to position and UMI to
                         identify possible duplicates [default=False]
 
-  input/output options:
+  Input/Output format options:
+    --in-format=IN_FORMAT
+                        File format of the input file. Format is usually
+                        implied from the extension of the filename, but maybe
+                        overridden with this option. Default=bam
+    --input-options=INPUT_OPTIONS
+                        Format string provided to htslib for reading. Mostly
+                        useful for CRAM formatted files. See samtools
+                        documentation
+    -i, --in-sam        [Deprecated] Input file is in sam format
+                        [default=False]
+    --reference-filename=REFERENCE_FILENAME
+                        File path or URL to the genome reference to be used
+                        when reading or writing CRAM files. By default, when
+                        reading a CRAM file, the reference recorded in the
+                        input file will be used unless this is specified. When
+                        writing, specifying a reference location is required.
+
+  Input/Output pipe options:
     -I FILE, --stdin=FILE
                         file to read stdin from [default = stdin].
     -L FILE, --log=FILE

tests/dedup_help

@@ -77,29 +77,35 @@ Options:
                         tags when selecting the best read amongst reads with
                         the same position and umi [default=none]
 
-  SAM/BAM options:
+  Input/Output format options:
     --in-format=IN_FORMAT
                         File format of the input file. Format is usually
                         implied from the extension of the filename, but maybe
                         overridden with this option. Default=bam
-    --out-format=OUT_FORMAT
-                        File format of the input file. Format is usually
-                        implied from the extension of the filename, but maybe
-                        overridden with this option. Default=bam
     --input-options=INPUT_OPTIONS
                         Format string provided to htslib for reading. Mostly
                         useful for CRAM formatted files. See samtools
                         documentation
+    -i, --in-sam        [Deprecated] Input file is in sam format
+                        [default=False]
+    --out-format=OUT_FORMAT
+                        File format of the input file. Format is usually
+                        implied from the extension of the filename, but maybe
+                        overridden with this option. Default=bam
     --output-options=OUTPUT_OPTIONS
                         Format string provided to htslib for writing. Mostly
                         useful for CRAM formatted files. See samtools
                         documentation
+    -o, --out-sam       [Deprecated] Output alignments in sam format
+                        [default=False]
     --reference-filename=REFERENCE_FILENAME
                         File path or URL to the genome reference to be used
                         when reading or writing CRAM files. By default, when
                         reading a CRAM file, the reference recorded in the
                         input file will be used unless this is specified. When
                         writing, specifying a reference location is required.
+
+  SAM/BAM filtering options:
     --mapping-quality=MAPPING_QUALITY
                         Minimum mapping quality for a read to be retained
                         [default=0]
@@ -108,9 +114,7 @@ Options:
                         read1, whether or not the template length is the same
     --chrom=CHROM       Restrict to one chromosome
     --subset=SUBSET     Use only a fraction of reads, specified by subset
-    -i, --in-sam        Input file is in sam format [default=False]
     --paired            paired input BAM. [default=False]
-    -o, --out-sam       Output alignments in sam format [default=False]
     --no-sort-output    Don't Sort the output
 
   Dedup and Count SAM/BAM options:
@@ -124,7 +128,7 @@ Options:
                         How to handle unpaired reads. Options are 'discard'or
                         'use' [default=use]
 
-  input/output options:
+  Input/Output pipe options:
     -I FILE, --stdin=FILE
                         file to read stdin from [default = stdin].
     -L FILE, --log=FILE

tests/extract_help

@@ -79,7 +79,7 @@ Options:
     --ignore-read-pair-suffixes
                         Ignore '\1' and '\2' read name suffixes
 
-  input/output options:
+  Input/Output pipe options:
     -I FILE, --stdin=FILE
                         file to read stdin from [default = stdin].
     -L FILE, --log=FILE

tests/group_help

@@ -80,29 +80,35 @@ Options:
                         tags when selecting the best read amongst reads with
                         the same position and umi [default=none]
 
-  SAM/BAM options:
+  Input/Output format options:
     --in-format=IN_FORMAT
                         File format of the input file. Format is usually
                         implied from the extension of the filename, but maybe
                         overridden with this option. Default=bam
-    --out-format=OUT_FORMAT
-                        File format of the input file. Format is usually
-                        implied from the extension of the filename, but maybe
-                        overridden with this option. Default=bam
     --input-options=INPUT_OPTIONS
                         Format string provided to htslib for reading. Mostly
                         useful for CRAM formatted files. See samtools
                         documentation
+    -i, --in-sam        [Deprecated] Input file is in sam format
+                        [default=False]
+    --out-format=OUT_FORMAT
+                        File format of the input file. Format is usually
+                        implied from the extension of the filename, but maybe
+                        overridden with this option. Default=bam
     --output-options=OUTPUT_OPTIONS
                         Format string provided to htslib for writing. Mostly
                         useful for CRAM formatted files. See samtools
                         documentation
+    -o, --out-sam       [Deprecated] Output alignments in sam format
+                        [default=False]
     --reference-filename=REFERENCE_FILENAME
                         File path or URL to the genome reference to be used
                         when reading or writing CRAM files. By default, when
                         reading a CRAM file, the reference recorded in the
                         input file will be used unless this is specified. When
                         writing, specifying a reference location is required.
+
+  SAM/BAM filtering options:
     --mapping-quality=MAPPING_QUALITY
                         Minimum mapping quality for a read to be retained
                         [default=0]
@@ -111,9 +117,7 @@ Options:
                         read1, whether or not the template length is the same
     --chrom=CHROM       Restrict to one chromosome
     --subset=SUBSET     Use only a fraction of reads, specified by subset
-    -i, --in-sam        Input file is in sam format [default=False]
     --paired            paired input BAM. [default=False]
-    -o, --out-sam       Output alignments in sam format [default=False]
     --no-sort-output    Don't Sort the output
 
   Group SAM/BAM options:
@@ -127,7 +131,7 @@ Options:
                         How to handle unpaired reads. Options are 'discard',
                         'use' or 'output' [default=use]
 
-  input/output options:
+  Input/Output pipe options:
     -I FILE, --stdin=FILE
                         file to read stdin from [default = stdin].
     -L FILE, --log=FILE

tests/umi_tools_help

@@ -4,7 +4,7 @@ For full UMI-tools documentation, see: https://umi-tools.readthedocs.io/en/lates
 umi_tools.py - Tools for UMI analyses
 =====================================
 
-:Author: Tom Smith & Ian Sudbury, CGAT
+:Author: Tom Smith & Ian Sudbery, CGAT
 :Tags: Genomics UMI
 
 There are 6 tools:
@@ -15,6 +15,7 @@ There are 6 tools:
   - dedup
   - count
   - count_tab
+  - prepare_for_em
 
 To get help on a specific tool, type:
 

tests/whitelist_help

@@ -69,7 +69,7 @@ Options:
     --ignore-read-pair-suffixes
                         Ignore '\1' and '\2' read name suffixes
 
-  input/output options:
+  Input/Output pipe options:
     -I FILE, --stdin=FILE
                         file to read stdin from [default = stdin].
     -L FILE, --log=FILE