How to scrape all comments from a YouTube video

YouTube comments are messy, but they're full of useful signal. Viewers say what confused them and describe products in language you would never get from a survey. They also have a knack for repeating the same complaint in wildly different words. Put all of that in one dataset and you can measure sentiment, monitor reactions to a competitor's videos, or turn recurring questions into your next content brief.

How a comment section is shaped

Public YouTube comment data has two levels that paginate separately. A top-level comment is posted directly on the video. That comment and its replies form a thread. A reply belongs to one top-level comment; replies do not nest further.

Each level hands you its own cursor. A page of comments carries pagination.next_cursor, which walks to the next page of threads. Each thread separately carries a replies_cursor, which opens that one thread's replies. A thread with no replies has a null replies_cursor, so you can skip it without spending a request.

{
  "video_id": "BXmOlCy0oBM",
  "returned_count": 20,
  "threads": [
    {
      "comment": { "comment_id": "Ugyf7z...", "text": "six seven", "author": "@strolol" },
      "replies_cursor": null
    }
  ],
  "pagination": {
    "next_cursor": "eyJ2IjoxLCJ2aWQiOiJCWG1PbEN5MG9CTSIs...",
    "completion": null
  }
}

YouTube's displayed comment count can include replies, so it will not match the number of top-level threads in the output. Keep thread and reply totals separate. Their sum is a useful sanity check, but it is not proof of completeness because the visible count can change while a long run is in progress.

The script

Seventy lines, no dependencies beyond requests, and it assumes the happy path: every call succeeds and the run finishes in one go. That is the version worth reading first. You need Python 3.9 or newer and a Tapline API key.

pip install requests
export TAPLINE_API_KEY=sk_live_...

import json
import os
import sys
import time

import requests

BASE = "https://api.tapline.sh/api/v1/youtube"
PAGE_SIZE = 20
PAUSE = 1.05  # stays inside the Free plan's 60 requests a minute

session = requests.Session()
session.headers["X-API-Key"] = os.environ["TAPLINE_API_KEY"]


def get(path, **params):
    response = session.get(f"{BASE}{path}", params=params, timeout=60)
    response.raise_for_status()
    time.sleep(PAUSE)
    return response.json()


def comment_pages(video_id):
    page = get(f"/videos/{video_id}/comments", sort="new", limit=PAGE_SIZE)
    while True:
        yield page
        cursor = page["pagination"]["next_cursor"]
        if not cursor:
            return
        page = get(f"/videos/{video_id}/comments", cursor=cursor, limit=PAGE_SIZE)


def all_replies(video_id, comment_id, cursor):
    replies = []
    while cursor:
        page = get(
            f"/videos/{video_id}/comments/{comment_id}/replies",
            cursor=cursor,
            limit=PAGE_SIZE,
        )
        replies.extend(page["replies"])
        cursor = page["pagination"]["next_cursor"]
    return replies


def scrape(video_id, out_path):
    threads = replies = 0
    completion = None

    with open(out_path, "w", encoding="utf-8") as out:
        for page in comment_pages(video_id):
            for thread in page["threads"]:
                comment = thread["comment"]
                comment["replies"] = all_replies(
                    video_id, comment["comment_id"], thread["replies_cursor"]
                )
                out.write(json.dumps(comment, ensure_ascii=False) + "\n")
                threads += 1
                replies += len(comment["replies"])

            completion = page["pagination"]["completion"]
            print(f"  {threads} threads, {replies} replies", end="\r")

    return threads, replies, completion


if __name__ == "__main__":
    video_id = sys.argv[1]
    threads, replies, completion = scrape(video_id, f"{video_id}.jsonl")
    print(f"\n{video_id}: {threads} threads, {replies} replies, completion={completion}")

The example video is Erlang: The Movie, whose comment section is small enough to finish in a few seconds.

$ python scrape_comments.py BXmOlCy0oBM
BXmOlCy0oBM: 27 threads, 12 replies, completion=exhausted

You get one JSONL row per top-level comment, with that comment's replies nested inside it. One row from the run above, trimmed to the fields most people use:

{
  "comment_id": "Ugwct29M55ab8twN5g54AaABAg",
  "text": "It's like a Monty Python sketch.",
  "like_count": 30,
  "published_at": "2022-08-01T18:05:46Z",
  "is_pinned": false,
  "author": "@Avicenna697",
  "author_is_uploader": false,
  "replies": [
    {
      "comment_id": "Ugwct29M55ab8twN5g54AaABAg.9SqcBVIIbte9fHaOv8VVud",
      "text": "...The wønderful telephøne system.",
      "author": "@jonaskoelker",
      "like_count": 12
    }
  ]
}

One line per thread keeps the output streamable for pandas, DuckDB, and jq, and a row is only written once that thread's replies are all in hand.

How it works

Top-level pagination

The first request sends sort=new and limit=20. Each later request sends the response's next_cursor back as cursor. The cursor already represents the video and sort order, so the script does not mix first-page parameters into later requests. A cursor from another walk may return a 400 response with an invalid_cursor error.

Reply pagination

Reply pagination starts with the replies_cursor attached to its parent thread. Threads with a null cursor need no reply request. Otherwise, the replies endpoint follows its own next_cursor until it is null. Each thread has a separate reply walk, so the distribution of replies across threads affects request count more than the total reply count alone.

Checking the result

The run above returned 27 top-level threads and 12 replies against a video that YouTube displayed as having 39 comments when this tutorial was checked. The totals agree, but the more reliable signal is pagination.completion on the last response. Both limit and sort settings can produce a terminal response without full coverage.

FAQ