How to tell youtube-comments-task to use Shadow Socks proxy

[zhirshya]

I just started to learn (node)js and want to scrape all comments of a list of youtube videos, by adding forEach loop and file output (for every video's comments) into your example code.
I got the following error for every youtube video id:

ERROR { component: 'fetch-first-page-token',
operation: 'fetch-first-page-token',
videoId: 'nPEYdw2Ssa8',
type: 'scraper-error',
message:
{ Error: connect ECONNREFUSED 8.7.198.45:443
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1159:14)
errno: 'ECONNREFUSED',
code: 'ECONNREFUSED',
syscall: 'connect',
address: '8.7.198.45',
port: 443 } }

I'm behind GFW of china, using SS proxy through brook GUI client on MacOS.
Could you please tell me how to tell youtube-comments-task to use the SS proxy?

[philbot9]

Hi there,

The easiest thing would be to modify the request opts in https://github.com/philbot9/youtube-comments-task/blob/master/src/lib/utils/request.js#L7
You will want to add your proxy configuration there. Refer to https://github.com/request/request

The right way to do this would be to add support for an options parameter in youtube-comments-task. I.e. fetchComments(videoId, pageToken, { proxy: '123.123.123.123:8888' }).
This package will eventually receive a major bump (see #3) so that option will likely have to wait until then.

cool lib, I've been using it. I ran into the same problem as this guy though and had to edit it to make it accept proxies. I recommend using classes or some kind of composition and dependency decoupling so you don't have to carry the "proxy" setting in every function.

this is what I've been doing but it's very hacky and doesn't work concurrently. in fetch-comments:

const req = require('./utils/request')
const fetchCommentsPromise = function (videoId, {pageToken, proxy} = {}) {
  // easiest way to add proxy to request without changing the whole code
  req.proxy = proxy

  return new Promise(function (resolve, reject) {
    if (!videoId) {
      reject('videoId parameter is required')
    }

    fetchComments(videoId, pageToken)
      .fork(reject, resolve)
  })
}

in utils/request

const request = opts =>
  new Task((rej, res) => {
    const optsWithJar = typeof opts === 'string'
      ? Object.assign({}, { jar: req.jar() }, { url: opts })
      : Object.assign({}, { jar: req.jar() }, opts)

    // easiest way to add proxy without changing entire code
    optsWithJar.proxy = request.proxy

    debug('sending request: %o', optsWithJar)

    req(optsWithJar, (err, response, body) => {
      if (err) {
        debug('request failed', err)
        rej(err)
      } else if (response.statusCode !== 200) {
        debug('request failed', response.statusCode)
        rej(`Request failed, Status ${response.statusCode}`)
      } else {
        res({ body, cookieJar: optsWithJar.jar })
      }
    })
  })