Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document how -u can accept multiple urls #10

Open
Oudwins opened this issue Aug 1, 2023 · 3 comments
Open

Document how -u can accept multiple urls #10

Oudwins opened this issue Aug 1, 2023 · 3 comments

Comments

@Oudwins
Copy link

Oudwins commented Aug 1, 2023

Is your feature request related to a problem? Please describe.
I am not sure if its -u url1.com url2.com or -u url1.com,url2.com or -u url1.com -u url2.com

I am sorry im not familiar enough with go yet to figure it out. But if you explain ill submit a pull request to update the readme

@twiny
Copy link
Owner

twiny commented Aug 15, 2023

Hey, I believe it should be -u url1.com -u url2.com.
Thank you for offering help with documentation that is really appreciated.
I'm working to release a new update, (branch: csv_url_reader) this will be update it.

@Oudwins
Copy link
Author

Oudwins commented Aug 15, 2023

Alright, when I have a little time i'll check that -u url1.com -u url2.com works. & try to make the PR if it does.

I am also in the process of learning go (previous js only andy hahaha), so might be able to contribute further.

Some of the things I would be interested in adding are:

  • I would love to split the writer/verifier & crawler in such a way that you can spawn as multiple of both separately.
  • I am not sure why but after running the same crawl for around 2-3 days it fills 1GB of ram on my pc. I imagine you are storing either the domains or the urls crawled in memory. Not sure the best solution for this, I had thought of moving those things to a SQLite or something similar

But first I got to learn go & i'm very busy with school so i'll do what I can.

@Oudwins
Copy link
Author

Oudwins commented Sep 27, 2023

Finally had time to test this

The code you suggest throws an error -> -u url1.com -u url2.com

This code -u url1.com url2.com does not throw an error but does not seem to work, it only picks up the first url

This code works -> -u url1.com,url2.com

Repository owner deleted a comment Jan 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants