How to log what GPTR does to a file?

[danieldekay]

I am using the GPTR module in my app, and I want the user to better understand what has happened while the report was written.
Is there a way to easily get the history of a report and save that to a file?

[ElishaKay]

Sup @danieldekay

I handled something similar a while back in the save to db PR, but we ended up going in another direction.

The main solution I pursued there was to add 2 more services to the docker-compose file:

• A Rabbitmq Server
• A Nodejs Server (which connects to Mongo & saves logs there)

If you wanted to take a similar approach, you could post a message to the rabbit-mq server somewhere within the stream_output function

Would be an interesting direction to revisit.

If the data is saved in mongodb, we could also easily add an API route to the NodeJS Server which would be called via the frontend & fetch the logs based on a given chat_id or task_id

[danieldekay]

Copilot suggested something easier:

        # Capture all logs generated while this line runs
        with open("researcher_log.txt", "w") as f:
            log_handler = logging.StreamHandler(f)
            log_handler.setLevel(logging.DEBUG)
            logger = logging.getLogger()
            logger.addHandler(log_handler)
            try:
                research_result = await researcher.conduct_research()
            finally:
                logger.removeHandler(log_handler)

[ElishaKay]

[danieldekay]

LOL. Indeed, sometimes.

In was already thinking "How can i make this work in concurrent requests, etc."
Or: could I hack into the stream_ouput function and make that work?

It kind of works, unless the job ends with an exception, because the exception is not logged.

[ElishaKay]

Another possible solution for users that want to handle gptr logs as they come, right within their python code, according to custom use cases:

from typing import Dict, Any
import asyncio
from gpt_researcher import GPTResearcher

class CustomLogsHandler:
    """A custom Logs handler class to handle JSON data."""
    def __init__(self):
        self.logs = []  # Initialize logs to store data

    async def send_json(self, data: Dict[str, Any]) -> None:
        """Send JSON data and log it."""
        self.logs.append(data)  # Append data to logs
        print(f"My custom Log: {data}")  # For demonstration, print the log

async def run():
    # Define the necessary parameters with sample values

    query = "What happened in the latest burning man floods?"
    report_type = "research_report"  # Type of report to generate
    report_source = "online"  # Could specify source like 'online', 'books', etc.
    tone = "informative"  # Tone of the report ('informative', 'casual', etc.)
    config_path = None  # Path to a config file, if needed

    # Initialize researcher with a custom WebSocket
    custom_logs_handler = CustomLogsHandler()

    researcher = GPTResearcher(
        query=query,
        report_type=report_type,
        report_source=report_source,
        tone=tone,
        config_path=config_path,
        websocket=custom_logs_handler
    )

    await researcher.conduct_research()  # Conduct the research
    report = await researcher.write_report()  # Write the research report

    return report

# Run the asynchronous function using asyncio
if __name__ == "__main__":
    asyncio.run(run())

Note: we should consider renaming the websocket parameter we pass into GPTR to make this use case more intuitive