improve content-fetch

[sywhb]

Some improvement on content-fetch:

1. remove scrapingbee
2. fix the sql to count saved items in database within a minute using created_at instead of saved_at as saved_at can be manually input
3. add an hour rate limiter to the API router which only allows 600 authenticated requests and 150 non-authenticated requests per hour on top of the minute rate limiter
4. hardcode some domains to block while fetching content and blocks a domain for a day if it was failed to fetch for 10 times in a day

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
omnivore-demo	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jul 25, 2024 5:13am
omnivore-prod	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jul 25, 2024 5:13am

[github-actions]

Squawk Report

✅ 0 violations across 1 file(s)

---

packages/db/migrations/0184.do.add_user_id_created_at_index_to_library_item.sql

-- Type: DO
-- Name: add_user_id_created_at_index_to_library_item
-- Description: Add an index of columns user_id and created_at to the library_item table for counting query

CREATE INDEX CONCURRENTLY IF NOT EXISTS library_item_user_id_created_at_idx ON omnivore.library_item (user_id, created_at);

✅ Rule Violations (0)

No violations found.

---

📚 More info on rules

⚡️ Powered by Squawk (1.1.2), a linter for PostgreSQL, focused on migrations

[sywhb]

Squawk Report

✅ 0 violations across 1 file(s)

packages/db/migrations/0184.do.add_user_id_created_at_index_to_library_item.sql

-- Type: DO
-- Name: add_user_id_created_at_index_to_library_item
-- Description: Add an index of columns user_id and created_at to the library_item table for counting query

CREATE INDEX CONCURRENTLY IF NOT EXISTS library_item_user_id_created_at_idx ON omnivore.library_item (user_id, created_at);

✅ Rule Violations (0)

No violations found.

📚 More info on rules

⚡️ Powered by Squawk (1.1.2), a linter for PostgreSQL, focused on migrations

@jacksonh I created an index on columns user_id and created_at on table library_item for the counting query

[jacksonh]

Can you explain this one?

[sywhb]

I replaced pessimistic_write with pessimistic_read lock strategy so it will prevent other transactions from modifying the data while it is being read but allows other transactions to read the data concurrently.

[sywhb]

@jacksonh

[jacksonh]

Actually a pretty big performance improvement too i think as the COUNT calls were adding up.

[jacksonh]

LGTM