Adding Git Embeds and other improvements #29

qxprakash · 2024-12-14T04:44:08Z

Key Features and Implementations:

1. Extended EmbedArgs:

Enhanced EmbedArgs to support five scenarios:

Git + Path: Clones the default branch and fetches the latest commit.
Git + Path + Branch: Clones a specified branch and fetches the latest commit.
Git + Path + Commit: Clones and checks out to a specific commit hash.
Git + Path + Tag: Clones and checks out to a specified tag.
Internal Nested Files: Previous flow intact

2. Improved Snippet Handling:

New Flow:

Entire files are now stored as snippets, enabling efficient parsing and embedding.
Supports embedding entire files directly when identifiers are not provided by the user.

Caching and Optimization:

Before cloning a repo, a unique file name is generated based on arguments (git, branch/tag/commit, path).
Snippets are saved in the .snippets directory.
File content hashes are compared with snippet hashes to prevent unnecessary updates.
Introduced DOCIFY_DISABLE_UPDATES to disable updates when unnecessary.

Edge Case Handling:

Added validation for file paths:
- Prohibited paths starting with ../.
- For Git URLs, paths starting with .. or / are forbidden.
Addressed Git URL edge cases by stripping .git suffixes to avoid hash mismatches.
Added helpful error messages for all edge cases.

let me know if any other changes are required.

With Regards
Prakash

…in that case throw an error that please pass the correct source file path

…remaining , edge cases remaining

… , currently switching to docif

…ly clones the default repo

…h , still need to handle the case where both branch and commit hash is provided

…t , either one of commit , branch , tag can only be passed with git , also added validation check that none of tag , commit , branch be passed without specifying git argument

…, it must be a file path

…ire crate process

… found don't clone

…, so in that case created snippet path without commit

…ck overflow error on the test tests::embed_args_tests::test_token_stream_conversion_git

optimized function , to get refs of only the latest commit

…S=1 or DOCIFY_DISABLE_UPDATES=1

sam0x17 · 2024-12-16T06:09:14Z

this looks great, I will add docs and merge this shortly!

qxprakash added 30 commits October 27, 2024 13:44

embed_internal_str with logs without making any changes

5155445

added more debug statements in embed internal string

e733fd7

added named embedargs

d7156d3

forbid use of "../" in file_path

7472229

added check where git url is passed but path starts with ".." or "/" …

cc1d56f

…in that case throw an error that please pass the correct source file path

clone working

a3b5e96

hash part working , branch , tag part remaining , network check part …

48e4f81

…remaining , edge cases remaining

added more logs

663b376

clone all the remote branch and switch to a particular branch working…

6430206

… , currently switching to docif

added branch_name optional param to clone_repo , if passed None it on…

36aea01

…ly clones the default repo

added commit hash when passed it check's out that specific commit has…

8a7e8dd

…h , still need to handle the case where both branch and commit hash is provided

tag working , now need to add that to the embedArgs

f95ff9d

added commit , branch , tag in embed args , along with validation tha…

6856250

…t , either one of commit , branch , tag can only be passed with git , also added validation check that none of tag , commit , branch be passed without specifying git argument

internet check added

f4f545e

added utility function for checking internet

f378780

network check working tested

11aebc3

positional arguments working

b1a88b5

added validation on first positional argument that it can't be a url …

66c171e

…, it must be a file path

tmp dir clean up but need to persist it till the lifecycle of the ent…

45f219d

…ire crate process

git dir cache key

62f01fd

changed "=" to ":" testing of all cases remaining

1b033e5

separate out functions , refactor

d67cef7

new flow working tested , prettify code and error messages remaining

1106f8e

git dir cache

ed70fa0

default branch working tested

e1148bc

removed oid from imports

1ffa78a

minimal flow working , caching case remaining where if the snippet is…

d8b425c

… found don't clone

caching working edge cases testing remaining

480aa92

moved creating of snppets dir to mod.rs

df82e6c

added edge case where commit was provided and internet not available …

73ab28a

…, so in that case created snippet path without commit

qxprakash added 27 commits November 2, 2024 02:48

hash of varied length working in case branch is not provided

fbbc823

added ident and tested

e78619c

changed parse for embed args , cleanup and addeed tests , getting sta…

1b3f34b

…ck overflow error on the test tests::embed_args_tests::test_token_stream_conversion_git

removed embed args test added logs in lib.rs

8af2054

cleanup and added comments

7d17b26

encapsulated git options into a separate function

4cc75ea

added missing imports

3f6c91c

get_snippet_file_name and existing_snippet_path functions

ae4daf0

pub fn get_remote_commit_sha_without_clone(

eaf1320

optimized function , to get refs of only the latest commit

commented out git refspec cloning progress

cf4849d

stripout .git from git_url it it's present

7363483

add check for empty string in file path in positional args

5a41628

changed has_internet to allow_updates , which will be false if DOCS_R…

92b994f

…S=1 or DOCIFY_DISABLE_UPDATES=1

added test 1 embed invalid args test failing

cce3497

updated tests

845da66

validate git url when passed as named args

70cc180

moved old file path validation checks into struct impl

9f2b7ac

used pattern matching in set_arg

3847fd5

removed the extra check

f7bf899

removed print statements from embed_internal_str

97cd401

removed all print statements from mod.rs

2de26c0

made changes to accomodate merge

7f29ff9

Merge branch 'staging'

0dfda07

fixes

f2178a6

all tests passing

1bf980c

fix warnings

1b696c0

Merge branch 'main' into staging

2d0612d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding Git Embeds and other improvements #29

Adding Git Embeds and other improvements #29

qxprakash commented Dec 14, 2024

sam0x17 commented Dec 16, 2024

Adding Git Embeds and other improvements #29

Are you sure you want to change the base?

Adding Git Embeds and other improvements #29

Conversation

qxprakash commented Dec 14, 2024

Key Features and Implementations:

1. Extended EmbedArgs:

2. Improved Snippet Handling:

New Flow:

Caching and Optimization:

Edge Case Handling:

sam0x17 commented Dec 16, 2024