NetTagCombine Reproduction

This project implements a Kaggle-oriented reproduction of cs224w-4-final.pdf using the StackExchange XML data schema.

Can Kaggle Download The Data Directly?

Yes, if the Kaggle notebook has Internet enabled. The Archive.org URLs are direct downloadable files, and the implementation streams XML from .7z archives instead of extracting full XML files to disk.

Required downloads:

• https://archive.org/download/stackexchange/stackoverflow.com-Posts.7z
• https://archive.org/download/stackexchange/stackoverflow.com-Comments.7z

Not required for this reproduction:

• Users.xml: user IDs needed here are already in posts/comments.
• Tags.xml: tag names are parsed from question Tags.
• Votes.xml, PostHistory.xml, PostLinks.xml, Badges.xml: not used by TagCombine or NetTagCombine as implemented from the paper.

Recommended Kaggle Run

Upload this tag_recommendations directory to Kaggle, or place it under /kaggle/working/tag_recommendations, then run:

cd /kaggle/working/tag_recommendations
pip install -r requirements-kaggle.txt
scripts/run_paper50k_from_archive_org.sh

The recommended script downloads and processes archives sequentially:

1. Download stackoverflow.com-Posts.7z.
2. Stream Posts.xml into SQLite, selecting the first 50,000 questions.
3. Delete Posts.7z by default to save disk.
4. Download stackoverflow.com-Comments.7z.
5. Stream relevant comments into the same SQLite DB.
6. Plot graph degree distributions.
7. Evaluate TagCombine and NetTagCombine.

Set DELETE_ARCHIVES_AFTER_USE=0 if you want to retain downloaded archives.

Outputs

Default output directory:

/kaggle/working/nettagcombine_paper50k

Important files:

• stack_overflow.sqlite: extracted questions, tags, answers, comments, and best-answer owners.
• plots/post_similarity_degree.png
• plots/user_interaction_degree.png
• plots/user_tag_bipartite_degree.png
• plots/post_similarity_communities.json
• results/metrics.csv
• results/metrics.json

Manual Commands

Download both required archives if disk permits:

scripts/download_stackexchange_archives.sh

Or download one archive at a time:

scripts/download_stackexchange_archives.sh stackoverflow.com-Posts.7z
scripts/download_stackexchange_archives.sh stackoverflow.com-Comments.7z

Extract posts and answers:

python -m nettagcombine.pipeline extract \
  --archives-dir /kaggle/working/stackexchange_archives \
  --db /kaggle/working/nettagcombine_paper50k/stack_overflow.sqlite \
  --max-questions 50000 \
  --top-tags 437 \
  --skip-comments

Extract comments after the posts/answers DB exists:

python -m nettagcombine.pipeline extract-comments \
  --archives-dir /kaggle/working/stackexchange_archives \
  --db /kaggle/working/nettagcombine_paper50k/stack_overflow.sqlite

Plot graph distributions:

python -m nettagcombine.pipeline plot-graphs \
  --db /kaggle/working/nettagcombine_paper50k/stack_overflow.sqlite \
  --out-dir /kaggle/working/nettagcombine_paper50k/plots \
  --post-sim-threshold 0.3 \
  --answer-score-threshold 2 \
  --write-communities

Evaluate:

python -m nettagcombine.pipeline evaluate \
  --db /kaggle/working/nettagcombine_paper50k/stack_overflow.sqlite \
  --out-dir /kaggle/working/nettagcombine_paper50k/results \
  --communities /kaggle/working/nettagcombine_paper50k/plots/post_similarity_communities.json \
  --folds 10 \
  --top-similar 50 \
  --weight-grid "0,0.2,0.4,0.6,0.8,1.0"

Implemented Data Schema

SQLite tables:

• questions: question id, owner user id, accepted answer id, creation date, score, title, body, preprocessed token text.
• question_tags: raw question tags parsed from StackExchange XML.
• answers: answer id, parent question id, answer owner user id, score.
• comments: comment id, post id, commenter user id, score.
• best_answers: accepted answer owner where available, otherwise highest score answer owner.
• tagspace: top 437 tags by frequency in the selected question set.

Implemented Models

TagCombine baseline:

• Binary Relevance with Multinomial Naive Bayes.
• Similarity ranking using top-50 TF-IDF cosine-similar training posts.
• Tag-term affinity based on tag/term co-occurrence.

NetTagCombine:

• Adds the network score from user-tag interactions.
• Uses question, answer, best-answer, and comment user IDs from the extracted StackExchange schema.
• Consumes communities from the post similarity graph and applies the paper's 80% community tag-representative rule when --communities is provided.

Limitations

• The Archive.org listing verified here is the current StackExchange dump, not the September 26, 2014 snapshot used by the CS224W paper.
• The paper used SNAP BigCLAM for overlapping communities. This implementation writes deterministic NetworkX label-propagation communities from the same post-similarity graph. If you produce SNAP BigCLAM communities separately, pass them as a JSON list of question-id lists via --communities.
• Full-dataset exact 10-fold evaluation is not recommended on standard Kaggle hardware. The code supports --max-questions 0, but the comments membership set and all-pairs similarity work will likely exceed normal notebook limits.