Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions advertools/crawlytics.py
Original file line number Diff line number Diff line change
Expand Up @@ -189,13 +189,13 @@
also use a regular expression to specify a set of columns. Here are some examples of
regular expressions that you might typically want to use:

* "img\_": Get all image columns, including all availabe `<img>` attributes.
* "jsonld\_": Get all JSON-LD columns.
* "resp_headers\_": Response headers.
* "request_headers\_": Request headers.
* "h\\\d": Heading columns, h1..h6.
* "redirect\_": Columns containing redirect information.
* "links\_": Columns containing link information.
* "img\\_": Get all image columns, including all availabe `<img>` attributes.
* "jsonld\\_": Get all JSON-LD columns.
* "resp_headers\\_": Response headers.
* "request_headers\\_": Request headers.
* "h\\d": Heading columns, h1..h6.
* "redirect\\_": Columns containing redirect information.
* "links\\_": Columns containing link information.

An important characteristic of these groups of columns is that you most likely don't
know how many they are, and what they might include, so a regular expression can save a
Expand Down
16 changes: 8 additions & 8 deletions advertools/robotstxt.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
==== =========== ================================= ================================== ========================= ================================= ================================
.. directive content etag robotstxt_last_modified robotstxt_url download_date
==== =========== ================================= ================================== ========================= ================================= ================================
0 User-agent \* "a850165d925db701988daf7ead7492d3" 2021-10-28 17:51:39+00:00 https://www.amazon.com/robots.txt 2022-02-11 19:33:03.200689+00:00
0 User-agent \\* "a850165d925db701988daf7ead7492d3" 2021-10-28 17:51:39+00:00 https://www.amazon.com/robots.txt 2022-02-11 19:33:03.200689+00:00
1 Disallow /exec/obidos/account-access-login "a850165d925db701988daf7ead7492d3" 2021-10-28 17:51:39+00:00 https://www.amazon.com/robots.txt 2022-02-11 19:33:03.200689+00:00
2 Disallow /exec/obidos/change-style "a850165d925db701988daf7ead7492d3" 2021-10-28 17:51:39+00:00 https://www.amazon.com/robots.txt 2022-02-11 19:33:03.200689+00:00
3 Disallow /exec/obidos/flex-sign-in "a850165d925db701988daf7ead7492d3" 2021-10-28 17:51:39+00:00 https://www.amazon.com/robots.txt 2022-02-11 19:33:03.200689+00:00
Expand Down Expand Up @@ -100,7 +100,7 @@
==== =========== =================================================================== ========================= ================================= ================================
.. directive content robotstxt_last_modified robotstxt_url download_date
==== =========== =================================================================== ========================= ================================= ================================
0 User-agent \* 2022-02-07 22:30:00+00:00 https://www.google.com/robots.txt 2022-02-11 19:52:13.375724+00:00
0 User-agent \\* 2022-02-07 22:30:00+00:00 https://www.google.com/robots.txt 2022-02-11 19:52:13.375724+00:00
1 Disallow /search 2022-02-07 22:30:00+00:00 https://www.google.com/robots.txt 2022-02-11 19:52:13.375724+00:00
2 Allow /search/about 2022-02-07 22:30:00+00:00 https://www.google.com/robots.txt 2022-02-11 19:52:13.375724+00:00
3 Allow /search/static 2022-02-07 22:30:00+00:00 https://www.google.com/robots.txt 2022-02-11 19:52:13.375724+00:00
Expand All @@ -109,7 +109,7 @@
290 comment \========================== NaT https://twitter.com/robots.txt 2022-02-11 19:52:13.461815+00:00
291 User-agent Googlebot NaT https://twitter.com/robots.txt 2022-02-11 19:52:13.461815+00:00
292 Allow /?_escaped_fragment_ NaT https://twitter.com/robots.txt 2022-02-11 19:52:13.461815+00:00
293 Allow /\*?lang= NaT https://twitter.com/robots.txt 2022-02-11 19:52:13.461815+00:00
293 Allow /\\*?lang= NaT https://twitter.com/robots.txt 2022-02-11 19:52:13.461815+00:00
397 comment Notice: Collection of data on Facebook through automated means is NaT https://facebook.com/robots.txt 2022-02-11 19:52:13.474456+00:00
398 comment prohibited unless you have express written permission from Facebook NaT https://facebook.com/robots.txt 2022-02-11 19:52:13.474456+00:00
399 comment and may only be conducted for the limited purpose contained in said NaT https://facebook.com/robots.txt 2022-02-11 19:52:13.474456+00:00
Expand Down Expand Up @@ -251,10 +251,10 @@
==== =================================== ============ ========== ===========
.. robotstxt_url user_agent url_path can_fetch
==== =================================== ============ ========== ===========
0 https://www.facebook.com/robots.txt \* / False
1 https://www.facebook.com/robots.txt \* /bbc False
2 https://www.facebook.com/robots.txt \* /groups False
3 https://www.facebook.com/robots.txt \* /hashtag/ False
0 https://www.facebook.com/robots.txt \\* / False
1 https://www.facebook.com/robots.txt \\* /bbc False
2 https://www.facebook.com/robots.txt \\* /groups False
3 https://www.facebook.com/robots.txt \\* /hashtag/ False
4 https://www.facebook.com/robots.txt Applebot / True
.. ... ... ... ...
75 https://www.facebook.com/robots.txt seznambot /hashtag/ True
Expand All @@ -277,7 +277,7 @@
==== =================================== =================== ========== ===========
.. robotstxt_url user_agent url_path can_fetch
==== =================================== =================== ========== ===========
0 https://www.facebook.com/robots.txt \* / False
0 https://www.facebook.com/robots.txt \\* / False
4 https://www.facebook.com/robots.txt Applebot / True
8 https://www.facebook.com/robots.txt Bingbot / True
12 https://www.facebook.com/robots.txt Discordbot / False
Expand Down
4 changes: 2 additions & 2 deletions advertools/youtube.py
Original file line number Diff line number Diff line change
Expand Up @@ -487,8 +487,8 @@ def search(
video game /m/01sjng Racing video game /m/0403l3g Role-playing
video game /m/021bp2 Simulation video game /m/022dc6 Sports game
/m/03hf_rm Strategy video game Sports topics /m/06ntj Sports
(parent topic) /m/0jm\_ American football /m/018jz Baseball
/m/018w8 Basketball /m/01cgz Boxing /m/09xp\_ Cricket /m/02vx4
(parent topic) /m/0jm\\_ American football /m/018jz Baseball
/m/018w8 Basketball /m/01cgz Boxing /m/09xp\\_ Cricket /m/02vx4
Football /m/037hz Golf /m/03tmr Ice hockey /m/01h7lh Mixed
martial arts /m/0410tth Motorsport /m/07bs0 Tennis /m/07_53
Volleyball Entertainment topics /m/02jjt Entertainment (parent
Expand Down