Adding CGA-CMV-Large to ConvoKit (#285)

laerdon · web-flow · commit 02868d92e67d · 2025-07-25T16:44:06.000-04:00
authored-by: laerdon
diff --git a/README.md b/README.md
@@ -66,8 +66,8 @@ These datasets can be downloaded using the `convokit.download()` [helper functio
 
 ### Conversations Gone Awry Datasets ([Wikipedia](https://convokit.cornell.edu/documentation/awry.html)/[CMV](https://convokit.cornell.edu/documentation/awry_cmv.html))
 
-Two related corpora of conversations that derail into antisocial behavior. One corpus (CGA-WIKI) consists of Wikipedia talk page conversations that derail into personal attacks as labeled by crowdworkers (4,188 conversations containing 30.021 comments). The other (CGA-CMV) consists of discussion threads on the subreddit ChangeMyView (CMV) that derail into rule-violating behavior as determined by the presence of a moderator intervention (6,842 conversations containing 42,964 comments).
-Name for download: `conversations-gone-awry-corpus` (for CGA-WIKI) or `conversations-gone-awry-cmv-corpus` (for CGA-CMV)
+Three related corpora of conversations that derail into antisocial behavior. One corpus (CGA-WIKI) consists of Wikipedia talk page conversations that derail into personal attacks as labeled by crowdworkers (4,188 conversations containing 30.021 comments). Another (CGA-CMV) consists of discussion threads on the subreddit ChangeMyView (CMV) that derail into rule-violating behavior as determined by the presence of a moderator intervention (6,842 conversations containing 42,964 comments). The last is a recent expansion of the CGA-CMV dataset, containing now 19,578 conversations and 116,793 utterances.
+Name for download: `conversations-gone-awry-corpus` (for CGA-WIKI), `conversations-gone-awry-cmv-corpus` (for CGA-CMV), and `conversations-gone-awry-cmv-corpus-large` (for CGA-CMV-Large)
 
 ### [Cornell Movie-Dialogs Corpus](https://convokit.cornell.edu/documentation/movie.html)
 
diff --git a/convokit/util.py b/convokit/util.py
@@ -36,9 +36,11 @@ def download(
         - "conversations-gone-awry-corpus": Wiki Personal Attacks Corpus
             Wikipedia talk page conversations that derail into personal attacks as labeled by crowdworkers
             (see http://www.cs.cornell.edu/~cristian/Conversations_gone_awry.html)
+        - "conversations-gone-awry-cmv-corpus-large"
+            Discussion threads on the subreddit ChangeMyView (CMV) that derail into rule-violating behavior, now updated with conversations up to 2022.
+            (see http://www.cs.cornell.edu/~cristian/Conversations_gone_awry.html)
         - "conversations-gone-awry-cmv-corpus"
             Discussion threads on the subreddit ChangeMyView (CMV) that derail into rule-violating behavior
-            (see http://www.cs.cornell.edu/~cristian/Conversations_gone_awry.html)
         -  "movie-corpus": Cornell Movie-Dialogs Corpus
             A large metadata-rich collection of fictional conversations extracted from raw movie scripts.
             (see https://www.cs.cornell.edu/~cristian/Chameleons_in_imagined_conversations.html)
diff --git a/docs/source/awry_cmv_large.rst b/docs/source/awry_cmv_large.rst
@@ -0,0 +1,81 @@
+Conversations Gone Awry Dataset (Large) - Reddit CMV version (CGA-CMV-Large)
+==============================================================
+
+A collection of conversations from the ChangeMyView (CMV) subreddit that derail into personal attacks (19,578 conversations, 116,793 comments), now updated with conversations up to 2022. Using this dataset, over the original version, is recommended.
+
+Summaries of conversation dynamics (SCDs) are available for a subset of the conversations.
+
+Distributed together with: Trouble on the Horizon: Forecasting the Derailment of Online Conversations as they Develop. Jonathan P. Chang and Crisitan Danescu-Niculescu-Mizil. EMNLP 2019.
+
+Summaries of conversation dynamics described in: How Did We Get Here? Summarizing Conversation Dynamics.  Yilun Hua, Nick Chernogor, Yuzhe Gu, Seoyon Julie Jeong, Miranda Luo, Cristian Danescu-Niculescu-Mizil. NAACL 2024.
+
+Example usage of the corpus and summaries: `SCD and Basic Examples <https://github.com/CornellNLP/ConvoKit/blob/master/examples/conversations-gone-awry-cmv/scd-example.ipynb>`_
+
+Dataset details
+---------------
+
+Speaker-level information
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Speakers in this dataset are Reddit users; their account names are taken as the user names.
+
+Utterance-level information
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Each utterance corresponds to a Reddit comment. For each utterance, we provide:
+
+* id: Reddit ID of the comment represented by the utterance
+* speaker: the speaker who authored the utterance
+* conversation_id: id of the first utterance in the conversation this utterance belongs to. Note that this differs from how 'conversation_id' is treated in ConvoKit's general Reddit corpora: in those corpora a conversation is considered to start with a Reddit post utterance, whereas in this corpus a conversation is considered to start with a top-level reply to a post.
+* reply_to: Reddit ID of the utterance to which this utterance replies to (None if the utterance represents a top-level comment, i.e., a reply to a post)
+* timestamp: time of the utterance
+* text: textual content of the utterance
+
+Metadata for each utterance is inherited from the general CMV corpus:
+
+* score: score (i.e., the number of upvotes minus the number of downvotes) of the content
+* top_level_comment: the id of the top level comment (None if the utterance is a post)
+* retrieved_on: unix timestamp of the time of when the data is retrieved
+* gilded: gilded status of the content
+* gildings: gilding information of the content
+* stickied: stickied status of the content
+* permalink: permanent link of the content
+* author_flair_text: flair of the author
+
+
+Conversational-level information
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Metadata for each conversation include:
+
+* pair_id: the id of the conversation that this conversation is paired with
+* has_removed_comment: whether the final comment in this thread was removed by CMV moderators for violation of Rule 2
+* split: which split (train, val, or test) this conversation was used in for the experiments described in "Trouble on the Horizon"
+* summary_meta: metadata related to conversation summaries, a list of dictionaries (one per summary available, possibly empty) with the following keys:
+    * summary_text: the text of the summary;
+    * summary_type: whether the summary is humman written by humans;(human_written_SCD) or generated automatically using the procedural prompt ("machine_generated_SCD") ;
+    * up_to_utterance_id: the last utterance considered when creating the summary;
+    * truncated_by: the number of utterances the transcript was truncated by when creating the summary (starting from the end);
+    * scd_split: whether the summary was in the train/test/validation split in the 2024 Summarizing Conversations Dynamics paper;
+
+
+Usage
+-----
+
+To download directly with ConvoKit:
+
+>>> from convokit import Corpus, download
+>>> corpus = Corpus(filename=download("conversations-gone-awry-cmv-corpus"))
+
+
+For some quick stats:
+
+>>> corpus.print_summary_stats()
+Number of Speakers: 24555
+Number of Utterances: 116793
+Number of Conversations: 19578
+
+Contact
+^^^^^^^
+
+Please email any questions to: cristian@cs.cornell.edu (Cristian Danescu-Niculescu-Mizil)
diff --git a/docs/source/datasets.rst b/docs/source/datasets.rst
@@ -4,6 +4,7 @@ Datasets
 .. toctree::
    Conversations Gone Awry Dataset - Wikipedia version (CGA-WIKI) <awry.rst>
    Conversations Gone Awry Dataset - Reddit CMV version (CGA-CMV) <awry_cmv.rst>
+   Conversations Gone Awry Dataset (Large) - Reddit CMV version (CGA-CMV-Large) <awry_cmv_large.rst>
    Cornell Movie-Dialogs Corpus <movie.rst>
    CANDOR Corpus <candor.rst>
    Parliament Question Time Corpus <parliament.rst>
diff --git a/docs/source/examples.rst b/docs/source/examples.rst
@@ -47,6 +47,7 @@ Forecaster
 ----------
 - `CRAFT forecasting of conversational derailment <https://github.com/CornellNLP/ConvoKit/blob/master/convokit/forecaster/CRAFT/demos/craft_demo_new.ipynb>`_
 - `Forecasting of conversational derailment using a cumulative bag-of-words model <https://github.com/CornellNLP/ConvoKit/blob/master/convokit/forecaster/tests/cumulativeBoW_demo.ipynb>`_
+- `Evaluating Transformer Fine-tuned Models (i.e. RoBERTa, Gemma) <https://github.com/CornellNLP/ConvoKit/blob/master/examples/forecaster/Run%20Transformer%20Fine-tuned%20Models.ipynb>`_
 
 Hyperconvo
 ----------
diff --git a/docs/source/forecaster.rst b/docs/source/forecaster.rst
@@ -79,3 +79,5 @@ These are subclasses of ForecasterModel, each implementing forecasting models us
 **Table 1: Forecasting derailment on CGA-CMV-large conversations.**
 The performance is measured in accuracy (Acc), precision (P), recall (R), F1, false positive rate (FPR), mean horizon (Mean H), and Forecast Recovery (Recovery) along with the correct and incorrect recovery rates. Results are reported as averages over five runs with
 different random seeds.
+
+For more information on how to produce a leaderboard string here, see the `Run Transformer Fine-tuned Models.ipynb <https://github.com/CornellNLP/ConvoKit/blob/master/examples/forecaster/Run%20Transformer%20Fine-tuned%20Models.ipynb>`_ notebook. If you would like to include your model in the leaderboard, please make a pull request adding the respective ForecasterModel and the version of the demo generating the leaderboard line. Please contact us on `Discord <https://discord.gg/R2ej9Kyr3K>`_ for assistance.
diff --git a/download_config.json b/download_config.json
diff --git a/examples/forecaster/Run Transformer Fine-tuned Models.ipynb b/examples/forecaster/Run Transformer Fine-tuned Models.ipynb
diff --git a/examples/forecaster/Transformer Forecaster demo.ipynb b/examples/forecaster/Transformer Forecaster demo.ipynb
