Architecture

[mikemonteith]

I just wanted to open up a ticket to discuss the architecture of this site.

As far as I see it:

1. A frontend (jekyll or similar static site generator)
2. Backend is github pages
3. Some way of generating transcripts from videos

On point 3:

An example of a site that does something very similar to this: https://iwantlistenthisrubbish.com/
It's all open source, so should be easy enough to copy the methodology, although I wonder if it would be easier to just get the transcripts directly out of youtube via it's API (https://developers.google.com/youtube/v3/docs/captions) I haven't tried to actually do this yet.

@davelab6 how did you generate the transcript that's already in the repo?

[gertfromthewell]

Hello! Found this via Discord. I actually started doing something a little bit like this a while ago (I wanted to play around with feeding them into a vector db and making a rag-based discord bot, but never quite found the time...)

I've got all the transcripts up to 27/2/2025 and would be happy to push them and the tooling I used to generate them if that's helpful.

Re: architecture, what's the intended use for this stuff? Having the transcripts all up on a site would be nice, but just having the transcript files all up on github (and maybe a machine-readable index file) might be a quicker/easier v1. Or at least, it'd be nice to have the raw transcripts all available in some easy to consume format as well.

[mikemonteith]

@gertfromthewell Please share!
The intended use for this site is to get the transcripts in a searchable format on the web in a user-friendly format, but I'm sure the raw transcripts could be useful for people too.
In my mind, we would have the raw transcripts in this repository, and then generate a UI from them.

[davelab6]

I just wanted to open up a ticket to discuss the architecture of this site.

As far as I see it:

1. A frontend (jekyll or similar static site generator)
2. Backend is github pages

Correct - to get started I thought I'd use the github hosted jekyll system, just as I had some familiarity with that, and it means not spending time with a compilation process via Github Actions (which is how any other static site generator can be used via Github Pages hosting).

3. Some way of generating transcripts from videos

The current site has a vtt-parser.js (permalink) which works for me locally to turn SRT ("SubRip Subtitle") format data into HTML, and my idea was to have the body of jeykll posts contain SRT data, and then the rest of the page's data as front-matter.

Last weekend I tried to figure out how to tell Github hosted Jekyll to turn off the entity conversion that turned --> into --> as I thought that was breaking the JS parser, instead of adjusting the JS to parse what it was outputting, which seemed like a dirty second choice. I thought I hadn't cracked it, but I just looked again and its working! lol

But a moment ago the live page showed WEBVTT Kind: captions Language: en 00:00:00.829 --> 00:00:06.150 This video is about Coronavirus and money. instead of the post-processed HTML, still, so I just realized the JS code was broken after all - and now fixed it, so the page is working.

On point 3:

An example of a site that does something very similar to this: iwantlistenthisrubbish.com It's all open source, so should be easy enough to copy the methodology, although I wonder if it would be easier to just get the transcripts directly out of youtube via it's API (developers.google.com/youtube/v3/docs/captions) I haven't tried to actually do this yet.

@davelab6 how did you generate the transcript that's already in the repo?

I added an INSTALL.txt to explain :)

[mikemonteith]

I have had a look at using the official youtube API to download the captions.
This works in theory, but the youtube API scopes are not very fine-grained, which means that the scope required to download captions is the same scope to be able to do everything! (Attached is a screenshot of the oAuth dialog).
The read-only scope does not allow access to download the captions for some reason.
This would open up an attack surface for someone to cause mischief with the channel.

One of the unofficial caption scrapers mentioned above might be better.

[davelab6]

Yeah I think given we are working on behalf of the channel we can use an alternative method. Personal opinion only, not the view of my employer.

[gertfromthewell]

@gertfromthewell Please share!

Have done!