Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ToDo list #484

Open
jarun opened this issue Nov 25, 2020 · 35 comments
Open

ToDo list #484

jarun opened this issue Nov 25, 2020 · 35 comments

Comments

@jarun
Copy link
Owner

jarun commented Nov 25, 2020

Continued from #343.

Notes

Please check the open issues for an area to contribute. bukuserver has several open features requests.

The list below is a growing one. While suggesting new features please consider contributing to Buku. The code is intentionally kept simple and easy to understand with comments. We'll be happy to assist any new contributor. We need your help!

Some of the fresh-baked features may not have been released yet. Grab the master branch for those.

Cooking

Identified tasks

  • support offline bookmark addition
  • support gemini protocol
  • support archive.is
  • bukuserver: disable favicon by default
  • bukuserver: bookmarklet documentation (see Bookmarklet #385)
  • bukuserver: thumbnail
  • bukuserver: add login feature (see Security features in bukuserver #309)
  • bukuserver: generate separate package (see Package bukuserver #307)
  • Import keywords from browser html (see Support for "real" keyword searches #311)
  • Android app (with the same schema) (probably a separate project)
  • Add option --preserve to ignore specific fields during auto-update [see issue Preserve some preexising data during update operation #327]
  • support Vivaldi browser
  • better XBEL compatibility
  • check for empty search results in piped operations
  • Test cases for exportdb() API (@rachmadaniHaryono)
  • Purge db file when all records are deleted
  • Fix tests, which are taking long time to finish (thanks @rachmadaniHaryono)
  • Import firefox exported json (title, url, tags, desc; parent folder if opted) [see API importdb()] (thanks @ckolumbus)
  • Disable fetch from web during auto-import, import and merge
  • Honor -n with -p (thanks @jpdasma)
  • Support search options with --export
  • Import "description" (into description field) and "keywards" (as tags) from HTML <meta> Tag
  • Result pagination
  • Convert bukuserver README from rst to markdown
  • Browse a bookmark (possibly dead URL) on Wayback machine
  • Show bookmarks to be deleted before deletion
  • Support keyword filtering (records having keywords a and b but not c and d) (thanks @saltyCatfish)
  • Support filtering by tags with search options (see Searching for tags and keywords #250) (thanks @saltyCatfish)
  • Port feature from googler/ddgr - omniprompt key O
  • Show results with most search keyword matches on top (thanks @mosegontar)
  • Text-mode user agent for Buku
  • Copy search result URL to cipboard
  • Read default Firefox profile name from profiles.ini (see Firefox auto-import breaks if profile name does not end with .default #212, thanks @alex-bender)
  • Support --format in search results (ref, thanks @mosegontar)
  • API documentation (comments need to be in NumPy format) (thanks @mosegontar)
  • Auto-import: optionally add parent folder name as tag, ask for unique tag [like importdb()]
  • Support custom colours (thanks @shv-q3)
  • Generate packages on Travis-CI using PackageCore (see Generate packages using PackageCore and Travis CI #189) (thanks @shaggytwodope)
  • Search multiple tags, exclusion in tag search (thanks @mosegontar)
  • Auto-import Firefox and Google Chrome bookmarks (thanks @alex-bender)
  • Suggest tags those go together
  • Append/overwrite/remove tags from prompt
  • Add more tests (ongoing activity @rachmadaniHaryono)
  • A browser plugin (thanks @samhh for bukubrow)
  • Text editor support (thanks @ZwodahS)
  • Need a PyPI maintainer (thanks @shaggytwodope)
  • Make refreshdb faster using threads (record updates should be synchronized)
  • Show usage count in tag list
  • Proxy support (thanks @denisfalqueto)
  • Continuous search at prompt
  • Add prompt help
  • Specify custom DB file to class BukuDb (library usage, no exposed option)
  • Move to urllib3
  • Handle redirects using referrer masking. Example URL. Fixed with urllib3.
  • Support URL shortening. This helps to share URLs. (see Support adding shortened URLs to the database #92 for limitations)
  • Make a bookmark title immutable via refreshdb()
  • Markdown import/export
  • Regex search
  • Ubuntu PPA (thanks @shaggytwodope)
  • Export specific tags to HTML
  • Exact word match using REGEX. Make substring match optional.
  • Delete all records based on a search result
  • Delete multiple items, support combination of indices and ranges
  • Append tags
  • Travis CI integration
  • Ubuntu deb package generation on new tag
  • Merge bookmark database files (for users who work on multiple systems)
  • Export bookmarks in FF or Chrome html format.
  • Option to add folder names as tags while importing HTML (thanks @Mohammadkhalifa)
  • Check and show upstream version
  • Anything else which would add value (please discuss in this thread)
@rachmadaniHaryono
Copy link
Collaborator

i just noticed that there is possibility to track user with favicon based on this https://github.com/jonasstrehle/supercookie

i may update bukuserver later to set favicon default to false.

@jarun
Copy link
Owner Author

jarun commented Feb 19, 2021

Sure thing!

@zz5zz
Copy link

zz5zz commented Aug 18, 2021

The bukuserver is an awesome frontend for Buku!

Some new functionality:

  • Add a Dark mode and possibility to turn on & off (create a custom CSS)
  • Bootstrap should be updated to minimum "bootstrap 3.4.1" to remove the known vulnerabilities (https://snyk.io/vuln/npm:jquery?lh=3.3.1)
  • jQuery should be updated to minimum "jquery 3.5.0" to remove the known vulnerabilities (https://snyk.io/vuln/npm:bootstrap?lh=3.3.5)
  • Add group / category to be used together with Tags to create a hierarchy when a lot of bookmarks have been added . That way you can as an example create a group called Newspapers with tags like electronics, legal, sports, economics etc.
  • Add year & month when added for both navigation purpose and stats over time (how many bookmarks was created in X)
  • When modal opens, should select the first input field by default
  • Add mousetrap.js to make keyboard shortcuts available like example CTRL+SHIFT+A to create a new bookmark

@jarun
Copy link
Owner Author

jarun commented Aug 19, 2021

@zz5zz are you interested in raising PRs?

@rachmadaniHaryono please review.

@rachmadaniHaryono
Copy link
Collaborator

@zz5zz

Add a Dark mode and possibility to turn on & off (create a custom CSS)
Bootstrap should be updated to minimum "bootstrap 3.4.1" to remove the known vulnerabilities (https://snyk.io/vuln/npm:jquery?lh=3.3.1)
jQuery should be updated to minimum "jquery 3.5.0" to remove the known vulnerabilities (https://snyk.io/vuln/npm:bootstrap?lh=3.3.5)

these all depend on flask-admin, but i do hope bukuserver can be updated to use latest flask before updating flask admin

Add group / category to be used together with Tags to create a hierarchy when a lot of bookmarks have been added . That way you can as an example create a group called Newspapers with tags like electronics, legal, sports, economics etc.

bukuserver depend on buku to handle tag so no group and category. but it is actually possible if bukuserver can parser tag and handle part of tag as category/group, for example programming: python tag will be under programming category group.

Add year & month when added for both navigation purpose and stats over time (how many bookmarks was created in X)

buku don't record created date

When modal opens, should select the first input field by default

this depend on flask admin and should be tested when flask admin updated

Add mousetrap.js to make keyboard shortcuts available like example CTRL+SHIFT+A to create a new bookmark

that seem to be nice addition, but it should be noted that bukuserver should handle external javascript dependency before adding new one . see also #307

@zz5zz
Copy link

zz5zz commented Aug 19, 2021

@zz5zz are you interested in raising PRs?

@rachmadaniHaryono please review.

I'll try to contribute by doing a PR. Very long time since I handled python so need to brush up a bit.

@jarun
Copy link
Owner Author

jarun commented Aug 19, 2021

Please take your time. Feel free to reach out for any help.

@RichardConroy
Copy link

What would it take to integrate the Buku database with Mac Spotlight search ? This strikes me as being a separate project, but its hard to know where to get started with it.

@jarun
Copy link
Owner Author

jarun commented Oct 29, 2021

Sorry, I am not a Mac user, no ideas.

@froschdesign
Copy link

@RichardConroy
You can use Alfred with a custom workflow. For the script you can use Python (or what you like) to execute the buku CLI command, the filtering of the bookmarks can be done by Alfred.

@GreenLunar
Copy link

GreenLunar commented Jan 4, 2022

Please add: Utilizing Bookie https://github.com/bookieio/Bookie

@rachmadaniHaryono
Copy link
Collaborator

i create #546 but after few new bukuserver issue i decided to convert it to draft and merge possible small issue into this pr

unfortunately i can't start weekend but hopefully next few days i can work on it

and maybe if possible upgrade bukusserver to newer flask version

@Remiol
Copy link

Remiol commented Jan 11, 2022

Buku is a great bookmark manager. I use it daily. I wish some of the related projects also work on windows like the bukubrow server and Oil. Search-as-you-type cli front-end. Maybe also an interface to gvim.
Anyway, thank you for the good work you've done.

@GreenLunar
Copy link

XMPP bot (also other IM platforms that provide API for bots)

@rachmadaniHaryono
Copy link
Collaborator

rachmadaniHaryono commented Jan 12, 2022

this is current pyproject.toml for buku

[tool.poetry]
name = "buku"
version = "4.6"
description = "Bookmark manager like a text-based mini-web."
authors = ["Arun Prakash Jana <[email protected]>"]
license = "GPLv3"
keywords = ["cli", "bookmarks", "tag", "utility"]
classifiers = [
  "Development Status :: 5 - Production/Stable",
  "Environment :: Console",
  "Intended Audience :: Developers",
  "Intended Audience :: End Users/Desktop",
  "License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
  "Natural Language :: English",
  "Operating System :: OS Independent",
  "Programming Language :: Python :: 3 :: Only",
  "Programming Language :: Python :: 3",
  "Programming Language :: Python :: 3.6",
  "Programming Language :: Python :: 3.7",
  "Programming Language :: Python :: 3.8",
  "Topic :: Internet :: WWW/HTTP :: Indexing/Search",
  "Topic :: Utilities",
]
homepage = "https://github.com/jarun/buku"
readme = "README.md"

[tool.poetry.dependencies]
python = ">=3.6,<4"
beautifulsoup4 = ">=4.4.1"
cryptography = ">=1.2.3"
html5lib = ">=1.0.1"
urllib3 = ">=1.23"
certifi = {version = "^2021.10.8", extras = ["ca-certificates"]}
appdirs = {version = ">=1.4.3", extras = ["server"]}
arrow = {version = ">=0.12.1", extras = ["server"]}
cffi = {version = ">=1.9.1", extras = ["server"]}
click = {version = ">=7.0", extras = ["server"]}
Flask-API = {version = ">=0.6.9", extras = ["server"]}
Flask-Bootstrap = {version = ">=3.3.7.1", extras = ["server"]}
flask-paginate = {version = ">=0.5.1", extras = ["server"]}
flask-reverse-proxy-fix = {version = ">=0.2.1", extras = ["server"]}
Flask-WTF = {version = ">=0.14.2", extras = ["server"]}
Flask = {version = ">=1.0.2,<2.0", extras = ["server"]}
idna = {version = ">=2.5", extras = ["server"]}
itsdangerous = {version = ">=0.24", extras = ["server"]}
Jinja2 = {version = ">=2.10.1", extras = ["server"]}
MarkupSafe = {version = ">=1.0", extras = ["server"]}
packaging = {version = ">=16.8", extras = ["server"]}
pyasn1 = {version = ">=0.2.3", extras = ["server"]}
pycparser = {version = ">=2.17", extras = ["server"]}
requests = {version = ">=2.21.0", extras = ["server"]}
six = {version = ">=1.10.0", extras = ["server"]}
Werkzeug = {version = ">=0.11.15,<0.15.0", extras = ["server"]}
flask-admin = {url = "https://github.com/flask-admin/flask-admin/archive/refs/heads/master.zip", extras=["server"]}
twine = {version = "^3.7.1", extras = ["packaging"]}

[tool.poetry.dev-dependencies]
attrs = ">=17.4.0"
flake8 = ">=3.4.1"
hypothesis = ">=6.0.0"
mypy-extensions = "0.4.1"
py = ">=1.5.0"
pylint = ">=1.7.2"
pytest-cov = "^3.0.0"
pytest-vcr = ">=1.0.2"
pytest = ">=6.2.1"
PyYAML = ">=4.2b1"
setuptools = ">=41.0.1"
vcrpy = ">=1.13.0"
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
[tool.poetry.scripts]
buku = 'buku:main'
bukuserver='bukuserver.server:cli'

the biggest change should be werkzeug for bukuserver

Werkzeug = {version = ">=0.11.15,<0.15.0", extras = ["server"]}

this is because flask_api require werkzeug._compat which only exist before v0.15.0

so therefore bukuserver packages that need to be upgraded

  • flask
  • flask-api
  • flask-admin
  • werkzeug
  • flask-reverse-proxy-fix

i'm still waiting pr for flask-reverse-proxy-fix

antarctica/flask-reverse-proxy-fix#1

if i finished with current bukuserver bugfix pr, i will fork the project and create new version

related issue

#449


why using poetry?

from their website

  • Dependency resolver: Poetry comes with an exhaustive dependency resolver, which will always find a solution if it exists.
    And get a detailed explanation if no solution exists.
  • Isolation: Poetry either uses your configured virtualenvs or creates its own to always be isolated from your system.
    The behavior is configurable.

while there is guideline, there is no entry about virtualenv and handling package requirements

predicted issue

  • buku have to be renamed back to buku.py
  • something wrong on pypi after release

to handle pypi release when this is accepted, maybe release alpha/beta/rc version before final version


there are also several tool that i want to add to project

  • commitizen: handle program version and commit message format
    this can be used for contributor, who want commit message format recommendation
  • pre-commit: check before commit
    there is several hook for this program that can be used for easier development
  • black: autoformat python code
    with this we can reduce guideline needed and let black format the code

with any of this tool is accepted, guideline on wiki should be changed as well


python 3.6 just ended on december 2021

https://endoflife.date/python


e:

follow nep 29 https://numpy.org/neps/nep-0029-deprecation_policy.html

idea: if poetry is accepted, post about recruiting new beginner developer for help on doc, test, and program

most of the time, beginner developer can see what is needed or what is not clear on buku

minimal requirement would be just understand poetry and git

e2:

add doc for using pipx to install buku

@rachmadaniHaryono
Copy link
Collaborator

@jarun, sorry i think i push a commit accidentally. c09ecfb

can you revert it? i think i can do it, but it is better if you do it.

sorry for this

@rachmadaniHaryono
Copy link
Collaborator

unrelated to above post

but in this issue i thought we could use yaml for issue form

#558 (comment)

imo i think we can update issue template so user have to fill buku version

https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/configuring-issue-templates-for-your-repository#creating-issue-forms

but i miss the notification on top of that page

Note: Issue forms are currently in beta for public repositories on GitHub.com only.

i thought buku should update its template because when you go to https://github.com/jarun/buku/blob/master/.github/ISSUE_TEMPLATE.md there is always this notification

You are using an old version of issue templates. Please update to the new issue template workflow. Learn more

so i create new issue template and feature request template from github

results

https://github.com/rachmadaniHaryono/Buku/issues/new/choose

https://github.com/rachmadaniHaryono/Buku/blob/master/.github/ISSUE_TEMPLATE/bug_report.md

---
name: Bug report
about: Create a report to help us improve
title: ''
labels: bug
assignees: ''

---

**Describe the bug**
A clear and concise description of what the bug is.
Before opening an issue, please try to reproduce on [the latest development version](https://github.com/jarun/Buku#from-source) first. The bug you noticed might have already been fixed.
If we need more information and there is no communication from the bug reporter within 7 days from the date of request, we will close the issue. If you have relevant information, resume discussion any time.

**To Reproduce**
Steps to reproduce the behavior:
1. ...
2. ...
3. ...
4. See error

Debug logs using the `-g` option

**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
 - Buku version: [e.g. 4.6, or commit hash]
 - OS: [e.g. windows, mac, linux, ubuntu]
 - Python version [e.g. 3.9, 3.10]
 - Terminal emulator and shell [e.g. "alacritty, zsh" or "gnome-terminal, fish"]
 - `locale` output, if relevant [e.g. LANG=en_US.UTF-8]

It's a good idea to set your locale to UFT-8. Please refer to [Buku #131](https://github.com/jarun/Buku/issues/30).

**Additional context**
Add any other context about the problem here.

https://github.com/rachmadaniHaryono/Buku/blob/master/.github/ISSUE_TEMPLATE/feature_request.md

---
name: Feature request
about: Suggest an idea for this project
title: ''
labels: ''
assignees: ''

---

**Is your feature request related to a problem? Please describe.**
Please consider contributing the feature back to `Buku` yourself. Feel free to discuss. We are more than happy to help.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.

looking at current template, i don't know if it is worth it to change it

@rachmadaniHaryono
Copy link
Collaborator

i tried to use poetry for buku not haven't found better configuration yet.

there is still issue to use it without copy buku to buku.py

i also kinda wary(?) about this because this should discussed with other packager


i also tried to create github page with sphinx on current doc, but it is harder

another possibility is to replace it with mkdocs, but that is last option

i am more familiarized with mkdocs for now, so maybe i will try it (and replace sphinx)


pre commit update pr is accepted, so it should be easier for contributor to automate some task

but contributing.md is not yet updated for this


unfortunately i don't give buku time for now, so no fixes for bukuserver for now

@apraga
Copy link

apraga commented Jul 14, 2022

Is there a built-in way to add a new tag from the prompt ?
From what I understand, g only works with existing tag.
Thanks !

@jarun
Copy link
Owner Author

jarun commented Jul 14, 2022

Use w to edit a bookmark and add a tag. You can add it to more bookmarks after that.

@apraga
Copy link

apraga commented Jul 14, 2022

Right, that’s what I’ve been doing. Here are 2 suggestions if other people are interested :

  • add a G option to add/modify/set tags by name
  • with this option, if the tag does not exist, create it and add it

@jarun
Copy link
Owner Author

jarun commented Jul 14, 2022

buku design doesn't allow having a tag that's not linked to at least one bookmark.

@rachmadaniHaryono
Copy link
Collaborator

rachmadaniHaryono commented Dec 3, 2022

…Speaking of running tests locally – one of test_network_handler_with_url runs (corresponding to a Google search query) always fails on my machine due to Google server responding with HTTP 429 Too Many Requests (I get the same response when sending the request with wget).

Not sure why this happens (perhaps it's related to me using a VPN for internet access?), but it certainly results in guaranteed fails every time I run unit tests locally. (…Does it really count as a unit test if the network requests are actually being sent? Normally you'd expect the requests themselves to be mocked or at least extracted to external functionality, in part to avoid such factors from affecting the outcome – after all, I doubt that the goal is to test specifically whether the code can communicate with Google servers.)

Does it really count as a unit test if the network requests are actually being sent?

@LeXofLeviafan

but it kinda also show that this function will not result the same thing on different computer, so it kinda work

but for consistency it is actually bad unit test

several options:

  1. have second alternative result instead of fail the test immediately
  2. cache the result, something similar to https://github.com/oprypin/pytest-golden
  3. mock the result

e: related #443

@LeXofLeviafan
Copy link
Collaborator

Unit tests are meant to test the code itself – whether the logic works correctly. And for a test to show that it "will not result in the same thing on different computer", it needs to be purposefully run on multiple machines (…also such a thing normally indicates that either the test or the code is broken 😅)

The reason it fails is purely incidental and completely external (i.e. this particular request has been blocked by Google for specific IPs or something like that) which has nothing to do with what the code is being tested for, really (unless the purpose is to test how various servers on the internet handle repeating requests or how their DDoS detection works).

…The caching thing looks like the way to go (provided the cache is actually included in the repo), except what you'd want is to cache the network requests instead of tests outputs. Ideally you'd want to generate such a cache file once (and never really need to modify it afterwards), and then mock the network environment in the test by providing stored response instead of live network feedback (or simply separate I/O from logic in the code and test the logic only). It can even be archived/compressed (a tarball?) to avoid having many small files or a large one in the repo (which seems to be the reason why #443 got rejected).

@LeXofLeviafan
Copy link
Collaborator

#644 does have a pull request though :-)

@rachmadaniHaryono
Copy link
Collaborator

@LeXofLeviafan you are correct, i will remove that

@LeXofLeviafan
Copy link
Collaborator

current issue with no active pr

I just realised that #648 appears to be missing from the list.

@rachmadaniHaryono
Copy link
Collaborator

added to refactor, thanks @LeXofLeviafan

@LeXofLeviafan
Copy link
Collaborator

API changes in #660:

  • bookmark data tuples returned from methods get_rec_all() & get_rec_by_id(), now have user-friendly properties (id, url, title, desc, tags/taglist, immutable; as well as for raw DB fields – tags_raw, flags)
  • methods get_rec_all(), list_using_id(), searchdb(), search_by_tag(), search_keywords_and_filter_by_tags() & exclude_results_from_search() are now guaranteed to return a list (empty if no data is found)
  • methods get_rec_id(), get_max_id() & add_rec() now return None as the "no ID" value
  • methods add_rec(), update_rec() & edit_update_rec() now treat the value of immutable parameter as a boolean (the default/noop value for update calls is None)
  • a FIELD_FILTER dictionary is introduced that contains fields formatting description; also, in format_json() (and print_json_safe()), the output format now matches the one described in CLI help

@LeXofLeviafan
Copy link
Collaborator

i just noticed that there is possibility to track user with favicon based on this https://github.com/jonasstrehle/supercookie

For the sake of completeness, I'll mention it here as well: the favicons in the app are downloaded from Google server and are treated as regular images, so their sites are not even aware of the user fetching them, let alone having means to make them individual per-user or persist through cache cleaning.

@LeXofLeviafan
Copy link
Collaborator

Suggestion: release this fork on PyPI (seeing as the original repo is stale), and change the [reverse_proxy] dependency accordingly (possibly reintegrating it back into the [server] deplist).

@sjehuda
Copy link
Contributor

sjehuda commented Jun 2, 2024

support offline bookmark addition

This seems to be done with argument --offline.

@LeXofLeviafan
Copy link
Collaborator

@jarun I've listed all pull-requests as items in the "Cooking" list (grouping when necessary); some might be considered too minor to list in the changelog though (at least individually).
Added the in-progress stuff as well (as unchecked items).

…Also, you may want check off some of the "Identified tasks" list items; I think some of them at least are done by now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

10 participants