I created a way to sync files between github and bolt.new however upon refresh it deletes the files i loaded

[TristanWillcox]

2024-11-22.15-29-02.mp4

in the video, you see I uploaded files from a previous bolt project I made on the official bolt website. Below is the code I use to transfer files from a GitHub repo into the bolt.new local version but as you can see in the video it deletes the files upon refresh and also doesn't not know the context of the code that is imported

help would be appreciated thanks

boltsyncfix.js

const https = require('node:https');
const readline = require('node:readline');
const querystring = require('node:querystring');
const fs = require('node:fs');

const parseGitHubUrl = (url) => {
  const match = url.match(/github\.com\/([^/]+)\/([^/]+)/);
  if (!match) throw new Error("Invalid GitHub URL format");
  return { owner: match[1], repo: match[2] };
};

const fetchData = (url, options) => {
  return new Promise((resolve, reject) => {
    const req = https.get(url, options, (res) => {
      let data = '';
      res.on('data', (chunk) => {
        data += chunk;
      });
      res.on('end', () => {
        if (res.statusCode >= 200 && res.statusCode < 300) {
          resolve(JSON.parse(data));
        } else {
          reject(new Error(data));
        }
      });
    });
    req.on('error', reject);
    req.end();
  });
};

const uploadToPastebin = async (content, apiKey) => {
  return new Promise((resolve, reject) => {
    const postData = querystring.stringify({
      api_dev_key: apiKey,
      api_option: 'paste',
      api_paste_code: content,
      api_paste_format: 'bash',
      api_paste_private: 1, // 0=public, 1=unlisted, 2=private
    });

    const options = {
      hostname: 'pastebin.com',
      path: '/api/api_post.php',
      method: 'POST',
      headers: {
        'Content-Type': 'application/x-www-form-urlencoded',
        'Content-Length': Buffer.byteLength(postData),
      },
    };

    const req = https.request(options, (res) => {
      let data = '';
      res.on('data', (chunk) => {
        data += chunk;
      });
      res.on('end', () => {
        if (res.statusCode === 200) {
          resolve(data); // Pastebin returns the paste URL on success
        } else {
          reject(new Error(`Pastebin API error: ${data}`));
        }
      });
    });

    req.on('error', (error) => reject(error));
    req.write(postData);
    req.end();
  });
};

const getRawFileLinks = async (url) => {
  try {
    const { owner, repo } = parseGitHubUrl(url);
    const branch = "main";
    const apiUrl = `https://api.github.com/repos/${owner}/${repo}/git/trees/${branch}?recursive=1`;

    const options = {
      headers: {
        'User-Agent': 'Node.js',
      },
    };

    const treeData = await fetchData(apiUrl, options);

    const files = treeData.tree.filter((item) => item.type === "blob");
    const rawLinks = files.map((file) => ({
      link: `https://raw.githubusercontent.com/${owner}/${repo}/${branch}/${file.path}`,
      path: file.path,
    }));

    const scriptContent = rawLinks
      .map(({ link, path }) => {
        const dir = path.includes('/') ? `mkdir -p "${path.substring(0, path.lastIndexOf('/'))}" && ` : '';
        return `${dir}curl -o "${path}" "${link}"`;
      })
      .join('\n');

    // const response = await uploadToPastebin(scriptContent, "");
    // console.log(response);
    console.log('Script written to creator.sh');
    fs.writeFileSync('creator.sh', scriptContent, 'utf-8');
    // if (response.toString().startsWith('https://pastebin.com/')) {
    //   const paste = response.toString().replace('https://pastebin.com/', 'https://pastebin.com/raw/');
    //   console.log("\x1b[32mTo install this repository on your local Bolt website run:\x1b[0m");
    //   console.log(`\x1b[33m curl -o creator.sh ${paste}\x1b[0m`);
    //   console.log("\x1b[30mTo clean all existing files run:\x1b[0m");
    //   console.log("\x1b[30m rm -rf *\x1b[0m");
    // }
  } catch (error) {
    console.error("Error:", error.message);
  }
};

const rl = readline.createInterface({
  input: process.stdin,
  output: process.stdout,
});

rl.question("Enter the GitHub repository URL: ", (url) => {
  getRawFileLinks(url).finally(() => rl.close());
});

[burgil]

This is very cool

Important Notes:

• I helped him make this script, feel free to ask questions, We are so close to load our project in there as can be seen in the video, please help if anyone knows how to fix it..
• We haven't really tested what happens for large projects or sub directories, so consider this a small patch rather than a full solution, use at your own risk
• if I didn't update here then we didn't find a way to fix it yet..
• Meanwhile I will dig into the source to try and find where is even the web containers and how to make them save or update...
• It uses GitHub free api with no token to pull the current files and create the commands needed to download each file and create folders in the web container
• The shorthand for pastebin did not work due to a socket error on the webcontainers, neither did the clean all existing files so this part is commented out, the content can be pasted directly anyways and it works as can be seen on the video, perhaps we can copy to clipboard directly instead of saving to a file.. so the pastebin part can be removed from the code
• We used https instead of axios so it can be ran directly using node boltsyncfix.js without needing any dependencies
• The existing Sync Files button seems to be one way, making changes or syncing with a directory that contain files did not seem to work
• I believe we tried all possibilities to get the UI to save before the refresh, it all comes down to something in the code now, we need to add a "Save" button or something that refreshes the memory of the AI because it "forgets" or "doesn't know" about the new changes we made using the terminal, We tried to refresh after forking messages and asking the AI to do something with the new code and Sync Files with an empty directory, it's all gone after the refresh..

[burgil]

I am still trying to solve this, forked the project, the updated changes and the clean script can be found below:

Reproduction Steps: (Tested on Win10)

git clone https://github.com/burgil/bolt.new-any-llm_windows
cd bolt.new-any-llm_windows

• Rename .env.local.rename to .env.local - No need to fill anything

npm install
npm start

• Open http://127.0.0.1:8788/
• Get an API key from some where for testing: e.g.
  
• Tell the AI hi to create an empty project
• Go to the terminal in the browser and paste the content created from our script (See below for updated instructions)

Use the script inside the repo:

• Updated Script: (Cleaned) https://github.com/burgil/bolt.new-any-llm_windows/blob/main/sync-fixer-bolt.cjs (required .cjs to signal commonjs to a module project if running with node)
• Updated Usage: cd bolt.new-any-llm_windows && npm run fix then paste a URL to generate your clone instructions in a text file e.g. Cloner_Synchr0nic_v3.txt
• Example project for the script (our project): https://github.com/Synchr0nic/v3
• Example script output: https://github.com/burgil/bolt.new-any-llm_windows/blob/main/Cloner_Synchr0nic_v3.txt

And the rest is still exactly like in the video, hope we can figure it out that will be awesome

[burgil]

I just figured out something... and I believe its the key to solve it!
When I refresh I noticed it literally reruns all the commands in the chat, but without actually using the AI, almost as if it uses a cached version, I confirmed this theory using the code below.. so what if when I make changes in the terminal they will also update the indexed db and the conversation state.. trying now to make it happen! 😁

• To test this theory I told the AI to make a new txt file and write in it hi
• And then I forked the chat once and refreshed
• The file was still there
• I told the AI to create another text file like this
• It created another file, I then refreshed and noticed the file is still there without needing to fork again
• I then proceeded to create a new file using the terminal: echo hi > hello3.txt
• I then told the AI to create hello4.txt
• I then asked the AI do you see hello3.txt? and it replied with yes
• I then refreshed and noticed the hello3.txt file is gone but the rest is still there
• I told the AI to come up with a random color and write it in a text file
• It wrote blue in color.txt and then I refreshed, still blue, it does not prompt the AI again, this will also be absurd behavior so instead it seems as if it caches the commands taken in the indexedDB and reruns it on a fresh webcontainer on reload.
• I explored all the storage in the devtools and found out about the usage of IndexedDB
• I wrote the following scripts in the devtool: (taken from the project itself)

async function openDatabase() {
  return new Promise((resolve) => {
    const request = indexedDB.open('boltHistory', 1);

    request.onupgradeneeded = (event) => {
      const db = (event.target).result;

      if (!db.objectStoreNames.contains('chats')) {
        const store = db.createObjectStore('chats', { keyPath: 'id' });
        store.createIndex('id', 'id', { unique: true });
        store.createIndex('urlId', 'urlId', { unique: true });
      }
    };

    request.onsuccess = (event) => {
      resolve((event.target).result);
    };

    request.onerror = (event) => {
      resolve(undefined);
    };
  });
}

const db = await openDatabase();

This defined db

Then:

    const transaction = db.transaction('chats', 'readwrite');
    const store = transaction.objectStore('chats');
    const chats = store.getAll();
    console.log(chats);

This will output the chats

Then I copied the chat I want to edit and pasted it in here:

    const transaction = db.transaction('chats', 'readwrite');
    const store = transaction.objectStore('chats');
    const add_request = store.put(HERE);

e.g...

    const transaction = db.transaction('chats', 'readwrite');
    const store = transaction.objectStore('chats');
    const add_request = store.put({
        "id": "5",
        "messages": [*Redacted*],
        "urlId": "5",
        "description": "Create Text File (fork)",
        "timestamp": "2024-11-22T..."
    });

In my example I modified this message from blue to green:

{
            "id": "Redacted",
            "role": "assistant",
            "content": "<boltArtifact id=\"create-hello6-file\" title=\"Create hello6 File with Color Name\">\n  <boltAction type=\"file\" filePath=\"hello6.txt\">\n    green\n  </boltAction>\n</boltArtifact>\n\nI have created a new file named \"hello6.txt\" with the content \"green\".",
            "createdAt": "2024-11-22T..."
        }

I refreshed, the file content changed from blue to green confirming my theory.

This will allow me to create a fix for our problem @TristanWillcox !

[burgil]

Done!

https://github.com/burgil/bolt.new-any-llm_windows

• Added a Clone button that works!
• Fixed cloning not saving after reload!
• New Backup/Restore Functions That Works! - Restore/Backup even when on the homepage
• Auto save/fork new chats
• Changed default model to Mistral Large and the default context size to 128000
• Auto open code area on reload
• Added create new project button
• Visual improvements

Feel free to close this issue.

2024-11-23_06-04-58.mp4

Update:

This is still doing some issues due to large context size, trying to optimize it now..

Fixed.

[nigelp]

This is super cool. Will you push this into the main oTToDev repo too? Also do you have to use Mistral for cloning? Thanks for the work. :)

[adriano2023666]

Hi everyone, I have a question. I accidentally deleted this code without knowing what it was for, and now I'm getting the following error:

vbnet
Copiar código
Application Error
NotFoundError: Failed to execute 'removeChild' on 'Node': The node to be removed is not a child of this node.
The deleted code is as follows:

jsx
Copiar código
const items = ['Item 1', 'Item 2', 'Item 3'];

return (

{items.map((item, index) => (
• {item}
// Replace index with a unique ID, if possible ))}

); I'm still learning, so I don't fully understand what's going on. Can anyone help me figure out how to fix this error? Erro de aplicação NotFoundError: Falha ao executar 'removeChild' em 'Node': O nó a ser removido não é filho deste nó. em removeChild (http://localhost:5173/node_modules/.vite/deps/chunk-TLFX46WQ.js?v=b7bbee02:15334:26) em commitDeletionEffectsOnFiber (http://localhost:5173/node_modules/.vite/deps/chunk-TLFX46WQ.js?v=b7bbee02:24388:21) em commitDeletionEffects (http://localhost:5173/node_modules/.vite/deps/chunk-TLFX46WQ.js?v=b7bbee02:24355:13) em recursivelyTraverseMutationEffects (http://localhost:5173/node_modules/.vite/deps/chunk-TLFX46WQ.js?v=b7bbee02:24552:17) em commitMutationEffectsOnFiber (http://localhost:5173/node_modules/.vite/deps/chunk-TLFX46WQ.js?v=b7bbee02:24615:15) em recursivelyTraverseMutationEffects (http://localhost:5173/node_modules/.vite/deps/chunk-TLFX46WQ.js?v=b7bbee02:24563:15) em commitMutationEffectsOnFiber (http://localhost:5173/node_modules/.vite/deps/chunk-TLFX46WQ.js?v=b7bbee02:24615:15) em recursivelyTraverseMutationEffects (http://localhost:5173/node_modules/.vite/deps/chunk-TLFX46WQ.js?v=b7bbee02:24563:15) em commitMutationEffectsOnFiber (http://localhost:5173/node_modules/.vite/deps/chunk-TLFX46WQ.js?v=b7bbee02:24615:15) em recursivelyTraverseMutationEffects (http://localhost:5173/node_modules/.vite/deps/chunk-TLFX46WQ.js?v=b7bbee02:24563:15)

[burgil]

This is super cool. Will you push this into the main oTToDev repo too? Also do you have to use Mistral for cloning? Thanks for the work. :)

Hi @nigelp, If the owner of the repo wants he may merge my changes, or you can use it directly from my repo
You don't have to use Mistral for cloning, Cloning simply asks for a GitHub url and then uses the GitHub api to fabricate the required mkdir and curl commands to download all the project, and then it generates messages for it in the AI conversation history (indexedDB)

Hi everyone, I have a question. I accidentally deleted this code without knowing what it was for, and now I'm getting the following error:

Hi @adriano2023666, I'm sorry but I don't know how to help you, your issue is not related to ours, try to submit a new issue detailing as much as possible what happened, where it happened, how to replicate it, and etc.