-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #41 from normand1/dn/refactor-and-sqlite-plugin
Refactor and sqlite plugin
- Loading branch information
Showing
81 changed files
with
1,623 additions
and
505 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,59 +1,27 @@ | ||
|
||
#! This is the configuration file for the podcast generator | ||
#! Ensure no passwords or API keys are stored in this file | ||
#! Passwords and API keys should be stored in .auth.env file which is ignored by git | ||
|
||
## Environment variables | ||
#articlesRSSFeedPlugin,podcastFeedPlugin | ||
PODCAST_DATA_SOURCE_PLUGINS=redditAPIPlugin | ||
#testerDataSourcePlugin | ||
SUBREDDIT=programming | ||
NUMBER_OF_POSTS_TO_FETCH=1 | ||
#testerDataSourcePlugin | ||
|
||
PODCAST_DATA_SOURCE_PLUGINS="sqliteTokenPlugin" | ||
|
||
PODCAST_INTRO_PLUGINS=standardIntroPlugin | ||
#testerIntroPlugin | ||
PODCAST_SCRAPER_PLUGINS=podcastRssAudioTranscriptScraper,newsStoryScraperPlugin | ||
#testerScraperPlugin | ||
PODCAST_SCRAPER_PLUGINS=tokenResearchScraperPlugin | ||
PODCAST_SEGMENT_WRITER_PLUGINS=topTenSegmentWriterPlugin | ||
#testerSegmentWriter | ||
PODCAST_OUTRO_PLUGINS=outroWriterPlugin | ||
#testerOutroPlugin | ||
PODCAST_PRODUCER_PLUGINS=producerPlugin | ||
PODCAST_FEEDS=https://feed.syntax.fm/ | ||
#https://podnews.net/rss | ||
# https://www.spreaker.com/show/6060119/episodes/feed | ||
ARTICLES_RSS_FEEDS=https://rss.app/feeds/_c2pBTHi1b82IDjwT.xml | ||
#,https://feed.syntax.fm/ | ||
NUMBER_OF_ITEMS_TO_FETCH=1 | ||
SHOULD_PAUSE_AND_VALIDATE_STORIES_BEFORE_SCRAPING=true | ||
|
||
# The following environment variables are for configuring the podcast for your specific needs | ||
# required | ||
PODCAST_NAME="Autonomous Tech Podcast" | ||
PODCAST_TYPE=tech | ||
PODCAST_DESCRIPTION="An autonomous tech podcast" | ||
|
||
# Langchain | ||
# LANGCHAIN_TRACING_V2=true | ||
# LANGCHAIN_ENDPOINT=https://api.smith.langchain.com | ||
# LANGCHAIN_PROJECT=pr-ajar-archives-11 | ||
|
||
# OpenAI | ||
OPENAI_MODEL_SUMMARY=gpt-4o-mini | ||
OPENAI_MAX_TOKENS_SUMMARY=4096 | ||
OPENAI_TEMPERATURE_SUMMARY=0.2 | ||
|
||
# Anthropic | ||
ANTHROPIC_MODEL=claude-3-5-sonnet-20240620 | ||
ANTHROPIC_MAX_TOKENS=8192 | ||
|
||
# The following environment variables are required to publish the podcast | ||
UPLOAD_TIMEOUT=60 * 5 * 1000 | ||
AUDIO_FILE_FORMAT='mp3' | ||
AUDIO_FILE_TEMPLATE='episode.%(ext)s' | ||
#PUPETEER_HEADLESS=false | ||
# SAVE_AS_DRAFT=0 | ||
|
||
# TTS Script Selection | ||
# TTS_SCRIPT=ttsLocalScript.sh | ||
# Uncomment the line below and comment out the line above to use ttsScript.sh instead | ||
TTS_SCRIPT=ttsLocalScript.sh | ||
PODCAST_RESEARCHER_PLUGINS=warpcastCastsResearcherPlugin,warpcastUserResearcherPlugin | ||
TOKEN_STORIES_DB_PATH=/Users/davidnorman/clanker-launch-bot/tokens.db | ||
TOKEN_STORIES_COUNT_LIMIT=1 | ||
|
||
SHOULD_PAUSE_AND_VALIDATE_STORIES_BEFORE_SCRAPING=false | ||
|
||
PODCAST_NAME="Slop Pod" | ||
PODCAST_TYPE="Memes and internet culture" | ||
PODCAST_DESCRIPTION="This is a podcast about the internet, memes and the people and stories behind it all" | ||
|
||
LLM_MODEL_PROVIDER=anthropic | ||
LLM_MODEL_VERSION_NAME=claude-3-5-sonnet-latest | ||
OPENAI_MAX_TOKENS_SUMMARY=2048 | ||
OPENAI_TEMPERATURE_SUMMARY=0.7 | ||
|
||
TTS_SCRIPT=ttsLocalScript.sh | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,65 @@ | ||
#! This is the configuration file for the podcast generator | ||
#! Ensure no passwords or API keys are stored in this file | ||
#! Passwords and API keys should be stored in .auth.env file which is ignored by git | ||
|
||
## Environment variables | ||
#articlesRSSFeedPlugin,podcastFeedPlugin | ||
PODCAST_DATA_SOURCE_PLUGINS=sqliteTokenPlugin | ||
#testerDataSourcePlugin | ||
SUBREDDIT=programming | ||
NUMBER_OF_POSTS_TO_FETCH=1 | ||
#testerDataSourcePlugin | ||
PODCAST_INTRO_PLUGINS=standardIntroPlugin | ||
#testerIntroPlugin | ||
PODCAST_SCRAPER_PLUGINS=tokenResearchScraperPlugin | ||
#testerScraperPlugin | ||
PODCAST_SEGMENT_WRITER_PLUGINS=topTenSegmentWriterPlugin | ||
#testerSegmentWriter | ||
PODCAST_OUTRO_PLUGINS=outroWriterPlugin | ||
#testerOutroPlugin | ||
PODCAST_PRODUCER_PLUGINS=producerPlugin | ||
PODCAST_FEEDS=https://feed.syntax.fm/ | ||
#https://podnews.net/rss | ||
# https://www.spreaker.com/show/6060119/episodes/feed | ||
ARTICLES_RSS_FEEDS=https://rss.app/feeds/_c2pBTHi1b82IDjwT.xml | ||
#,https://feed.syntax.fm/ | ||
NUMBER_OF_ITEMS_TO_FETCH=1 | ||
SHOULD_PAUSE_AND_VALIDATE_STORIES_BEFORE_SCRAPING=true | ||
|
||
# The following environment variables are for configuring the podcast for your specific needs | ||
# required | ||
PODCAST_NAME="Slop Cast" | ||
PODCAST_TYPE="crypto, tech, memes" | ||
PODCAST_DESCRIPTION="A podcast about the latest memecoins on farcaster and other platforms" | ||
|
||
# Token Stories | ||
TOKEN_STORIES_DB_PATH=/Users/davidnorman/clanker-launch-bot/tokens.db | ||
|
||
# Langchain | ||
# LANGCHAIN_TRACING_V2=true | ||
# LANGCHAIN_ENDPOINT=https://api.smith.langchain.com | ||
# LANGCHAIN_PROJECT=pr-ajar-archives-11 | ||
|
||
# anthropic or openai | ||
LLM_MODEL_PROVIDER=anthropic | ||
|
||
# OpenAI | ||
LLM_MODEL_VERSION_NAME=gpt-4o-mini | ||
OPENAI_MAX_TOKENS_SUMMARY=4096 | ||
OPENAI_TEMPERATURE_SUMMARY=0.2 | ||
|
||
# Anthropic | ||
ANTHROPIC_MODEL=claude-3-5-sonnet-latest | ||
ANTHROPIC_MAX_TOKENS=8192 | ||
|
||
# The following environment variables are required to publish the podcast | ||
UPLOAD_TIMEOUT=60 * 5 * 1000 | ||
AUDIO_FILE_FORMAT='mp3' | ||
AUDIO_FILE_TEMPLATE='episode.%(ext)s' | ||
#PUPETEER_HEADLESS=false | ||
# SAVE_AS_DRAFT=0 | ||
|
||
# TTS Script Selection | ||
# TTS_SCRIPT=ttsLocalScript.sh | ||
# Uncomment the line below and comment out the line above to use ttsScript.sh instead | ||
TTS_SCRIPT=ttsLocalScript.sh |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,73 +1,17 @@ | ||
#!/bin/zsh | ||
|
||
# Prompt user for configuration choice | ||
echo -n "Enter configuration choice\n (n) news | (p) podcasts | (nl) newsletter): " | ||
read choice | ||
cd podcastTextGenerationApp/charts | ||
|
||
# Define the new values based on the chosen configuration | ||
if [[ "$choice" == "news" || "$choice" == "n" ]]; then | ||
NEW_PODCAST_DATA_SOURCE_PLUGINS=hackerNewsAPIPlugin | ||
NEW_PODCAST_INTRO_PLUGINS=standardIntroPlugin | ||
NEW_PODCAST_SCRAPER_PLUGINS=newsStoryScraperPlugin | ||
NEW_PODCAST_SUMMARY_PLUGINS=storySummaryPlugin | ||
NEW_PODCAST_SEGMENT_WRITER_PLUGINS=topTenSegmentWriterPlugin | ||
NEW_PODCAST_OUTRO_PLUGINS=outroWriterPlugin | ||
NEW_PODCAST_PRODUCER_PLUGINS=producerPlugin | ||
elif [[ "$choice" == "podcasts" || "$choice" == "p" ]]; then | ||
echo -n "Enter the podcast RSS Feed: " | ||
read podcast_feed | ||
if grep -q "PODCAST_FEEDS=" podcastTextGenerationApp/podcastDataSourcePlugins/.env.datasource; then | ||
# If the variable exists in the file, replace it | ||
sed -i '' "s#^PODCAST_FEEDS=.*#PODCAST_FEEDS=$podcast_feed#" podcastTextGenerationApp/podcastDataSourcePlugins/.env.datasource | ||
else | ||
# If the variable doesn't exist in the file, add it | ||
echo "PODCAST_FEEDS=$podcast_feed" >> podcastTextGenerationApp/podcastDataSourcePlugins/.env.datasource | ||
fi | ||
echo -n "Enter the number of items to fetch: " | ||
read number_of_items | ||
if grep -q "NUMBER_OF_ITEMS_TO_FETCH=" podcastTextGenerationApp/podcastDataSourcePlugins/.env.datasource; then | ||
# If the variable exists in the file, replace it | ||
sed -i '' "s#^NUMBER_OF_ITEMS_TO_FETCH=.*#NUMBER_OF_ITEMS_TO_FETCH=$number_of_items#" podcastTextGenerationApp/podcastDataSourcePlugins/.env.datasource | ||
else | ||
# If the variable doesn't exist in the file, add it | ||
echo "NUMBER_OF_ITEMS_TO_FETCH=$number_of_items" >> podcastTextGenerationApp/podcastDataSourcePlugins/.env.datasource | ||
fi | ||
NEW_PODCAST_DATA_SOURCE_PLUGINS=podcastFeedPlugin | ||
NEW_PODCAST_INTRO_PLUGINS=standardIntroPlugin | ||
NEW_PODCAST_SCRAPER_PLUGINS=rawScraperPlugin | ||
NEW_PODCAST_SUMMARY_PLUGINS=storySummaryPlugin | ||
NEW_PODCAST_SEGMENT_WRITER_PLUGINS=topTenSegmentWriterPlugin | ||
NEW_PODCAST_OUTRO_PLUGINS=outroWriterPlugin | ||
NEW_PODCAST_PRODUCER_PLUGINS=producerPlugin | ||
elif [[ "$choice" == "newsletter" || "$choice" == "nl" ]]; then | ||
echo -n "Enter the newsletter RSS Feed: " | ||
read newsletter_feed | ||
if grep -q "NEWSLETTER_RSS_FEEDS=" podcastTextGenerationApp/podcastDataSourcePlugins/.env.datasource; then | ||
# If the variable exists in the file, replace it | ||
sed -i '' "s#^NEWSLETTER_RSS_FEEDS=.*#NEWSLETTER_RSS_FEEDS=$newsletter_feed#" podcastTextGenerationApp/podcastDataSourcePlugins/.env.datasource | ||
else | ||
# If the variable doesn't exist in the file, add it | ||
echo "NEWSLETTER_RSS_FEEDS=$newsletter_feed" >> podcastTextGenerationApp/podcastDataSourcePlugins/.env.datasource | ||
fi | ||
NEW_PODCAST_DATA_SOURCE_PLUGINS=articlesRSSFeedPlugin | ||
NEW_PODCAST_INTRO_PLUGINS=standardIntroPlugin | ||
NEW_PODCAST_SCRAPER_PLUGINS=rssItemScraperPlugin | ||
NEW_PODCAST_SUMMARY_PLUGINS=storySummaryPlugin | ||
NEW_PODCAST_SEGMENT_WRITER_PLUGINS=topTenSegmentWriterPlugin | ||
NEW_PODCAST_OUTRO_PLUGINS=outroWriterPlugin | ||
NEW_PODCAST_PRODUCER_PLUGINS=producerPlugin | ||
# Generate templates | ||
echo "Generating templates..." | ||
if helmfile template > output.yaml 2>/dev/tty; then | ||
# Extract environment variables for validation | ||
grep -A 1000 "config.env:" output.yaml | sed 's/^ *//' | sed '1d' > ../../.config.env | ||
cd ../../ | ||
|
||
# Source the updated environment variables | ||
source .config.env | ||
echo "Plugins configured! 🚀🚀🚀" | ||
else | ||
echo "Invalid choice. Exiting." | ||
exit 1 | ||
echo "Failed to generate templates. Please check for errors." | ||
fi | ||
|
||
# Update the .config.env file | ||
sed -i '' "s/^PODCAST_DATA_SOURCE_PLUGINS=.*/PODCAST_DATA_SOURCE_PLUGINS=$NEW_PODCAST_DATA_SOURCE_PLUGINS/" .env | ||
sed -i '' "s/^PODCAST_INTRO_PLUGINS=.*/PODCAST_INTRO_PLUGINS=$NEW_PODCAST_INTRO_PLUGINS/" .env | ||
sed -i '' "s/^PODCAST_SCRAPER_PLUGINS=.*/PODCAST_SCRAPER_PLUGINS=$NEW_PODCAST_SCRAPER_PLUGINS/" .env | ||
sed -i '' "s/^PODCAST_SUMMARY_PLUGINS=.*/PODCAST_SUMMARY_PLUGINS=$NEW_PODCAST_SUMMARY_PLUGINS/" .env | ||
sed -i '' "s/^PODCAST_SEGMENT_WRITER_PLUGINS=.*/PODCAST_SEGMENT_WRITER_PLUGINS=$NEW_PODCAST_SEGMENT_WRITER_PLUGINS/" .env | ||
sed -i '' "s/^PODCAST_OUTRO_PLUGINS=.*/PODCAST_OUTRO_PLUGINS=$NEW_PODCAST_OUTRO_PLUGINS/" .env | ||
sed -i '' "s/^PODCAST_PRODUCER_PLUGINS=.*/PODCAST_PRODUCER_PLUGINS=$NEW_PODCAST_PRODUCER_PLUGINS/" .env | ||
|
||
echo "Configuration updated successfully." |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
apiVersion: v2 | ||
name: mypodcast | ||
description: A helm chart to generate a .config.env for a publication | ||
type: application | ||
version: 0.1.0 | ||
appVersion: "1.0" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
# Uncomment the plugins you want to use and update their values in the associated yaml file as needed | ||
releases: | ||
- name: hyperfeeder | ||
namespace: default | ||
chart: ./ | ||
values: | ||
- values/base.yaml | ||
- values/researchers.yaml | ||
- values/podcast.yaml | ||
- values/llm.yaml | ||
- values/tts.yaml | ||
- values/sqliteToken.yaml | ||
# - values/reddit.yaml | ||
# - values/podcastFeed.yaml | ||
# - values/articlesRSS.yaml | ||
# - values/newsletterRSS.yaml | ||
# - values/hackerNewsAPI.yaml | ||
# - values/arxivApi.yaml | ||
|
Oops, something went wrong.