Add more bot platforms #542

luciomartinez · 2023-11-28T10:02:15Z

Hey,

I thought it might be helpful to have a few more popular bots besides Google bot (already registered as a platform).

This list is not extensive, yet it includes the most popular ones worldwide, including the main Russian and Chinese search engines.

One of the down sides of having this list is its maintenance, so feel free to skip it from merging if you deem it so.

Thanks,
Lucio.

lancedikson · 2024-07-23T08:16:27Z

Hi @luciomartinez! Sorry it took a while to get back to your PR. If you still available, can you please also add tests for these platforms so that we have examples in the tests?

luciomartinez · 2024-08-21T11:44:32Z

src/parser-platforms.js

+
+  /* Baidu */
+  {
+    test: [/baiduspider/i],


https://www.baidu.com/search/robots_english.html

luciomartinez · 2024-08-21T11:44:55Z

src/parser-platforms.js

+
+  /* Bingbot */
+  {
+    test: [/bingbot/i],


https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0

luciomartinez · 2024-08-21T11:49:10Z

src/parser-platforms.js

+
+  /* DuckDuckBot */
+  {
+    test: [/duckduckbot/i],


https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot/

luciomartinez · 2024-08-21T11:56:19Z

src/parser-platforms.js

@@ -18,6 +18,105 @@ export default [
    },
  },

+  /* AmazonBot */
+  {
+    test: [/Amazonbot/i],


https://developer.amazon.com/amazonbot

luciomartinez · 2024-08-21T11:56:38Z

src/parser-platforms.js

+
+  /* Internet Archive Crawler */
+  {
+    test: [/ia_archiver/i],


https://web.archive.org/web/20141022124819/http://archive.org:80/about/exclude.php

luciomartinez · 2024-08-21T11:57:09Z

src/parser-platforms.js

+
+  /* Meta Web Crawler */
+  {
+    test: [/facebookexternalhit/i, /facebookcatalog/i],


https://developers.facebook.com/docs/sharing/webmasters/web-crawlers

luciomartinez · 2024-08-21T12:01:15Z

src/parser-platforms.js

+
+  /* Yahoo! Slurp */
+  {
+    test: [/yahoo/i],


https://help.yahoo.com/kb/SLN22600.html

luciomartinez · 2024-08-21T12:04:34Z

src/parser-platforms.js

+
+  /* Yandex */
+  {
+    test: [/yandexbot/i, /yandexmobilebot/i],


https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html

luciomartinez · 2024-08-21T12:08:08Z

src/parser-platforms.js

+
+  /* Pingdom */
+  {
+    test: [/pingdom/i],


Best guest I suppose 🙂

luciomartinez · 2024-08-21T12:26:22Z

CI logs don't show exactly which test is failing.

Trying to build it locally, I get this error:

$ npm run build

> [email protected] build
> webpack --config webpack.config.js

node:internal/crypto/hash:79
  this[kHandle] = new _Hash(algorithm, xofLen, algorithmId, getHashCache());
                  ^

Error: error:0308010C:digital envelope routines::unsupported
    at new Hash (node:internal/crypto/hash:79:19)
    at Object.createHash (node:crypto:139:10)
    at module.exports (/workspaces/bowser/node_modules/webpack/lib/util/createHash.js:135:53)
    at NormalModule._initBuildHash (/workspaces/bowser/node_modules/webpack/lib/NormalModule.js:417:16)
    at /workspaces/bowser/node_modules/webpack/lib/NormalModule.js:452:10
    at /workspaces/bowser/node_modules/webpack/lib/NormalModule.js:323:13
    at /workspaces/bowser/node_modules/loader-runner/lib/LoaderRunner.js:367:11
    at /workspaces/bowser/node_modules/loader-runner/lib/LoaderRunner.js:233:18
    at context.callback (/workspaces/bowser/node_modules/loader-runner/lib/LoaderRunner.js:111:13)
    at /workspaces/bowser/node_modules/babel-loader/lib/index.js:55:103 {
  opensslErrorStack: [
    'error:03000086:digital envelope routines::initialization error',
    'error:0308010C:digital envelope routines::unsupported'
  ],
  library: 'digital envelope routines',
  reason: 'unsupported',
  code: 'ERR_OSSL_EVP_UNSUPPORTED'
}

Node.js v20.16.0

I'm not sure what I'm doing wrong here 😕

Add bot platforms

e417ea2

luciomartinez commented Aug 21, 2024

View reviewed changes

Add Facebook and IA crawler

d197a44

luciomartinez commented Aug 21, 2024

View reviewed changes

src/parser-platforms.js

/* Yahoo! Slurp */

{

test: [/yahoo/i],

Copy link

Contributor Author

luciomartinez Aug 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://help.yahoo.com/kb/SLN22600.html

Add mobile layout of Yandex bot

d668404

luciomartinez commented Aug 21, 2024

View reviewed changes

src/parser-platforms.js

/* Pingdom */

{

test: [/pingdom/i],

Copy link

Contributor Author

luciomartinez Aug 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Best guest I suppose 🙂

Add tests

6647d3b

Fix typo

1e86c2e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add more bot platforms #542

Add more bot platforms #542

luciomartinez commented Nov 28, 2023

lancedikson commented Jul 23, 2024

luciomartinez Aug 21, 2024 •

edited

Loading

luciomartinez Aug 21, 2024

luciomartinez Aug 21, 2024

luciomartinez Aug 21, 2024

luciomartinez Aug 21, 2024

luciomartinez Aug 21, 2024

luciomartinez Aug 21, 2024

luciomartinez Aug 21, 2024

luciomartinez Aug 21, 2024

luciomartinez commented Aug 21, 2024 •

edited

Loading

Add more bot platforms #542

Are you sure you want to change the base?

Add more bot platforms #542

Conversation

luciomartinez commented Nov 28, 2023

lancedikson commented Jul 23, 2024

luciomartinez Aug 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luciomartinez commented Aug 21, 2024 • edited Loading

luciomartinez Aug 21, 2024 •

edited

Loading

luciomartinez commented Aug 21, 2024 •

edited

Loading