Advanced: Set up a GroupMe bot that can read websites

General GroupMe
, , ,
1

Build a Conversational AI Bot for GroupMe with a Live Web Data Feed

Goal: To build a versatile, conversational GroupMe bot that uses live data scraped from any website as its knowledge base. Instead of simple commands, this bot engages with any user message, fetches fresh data in real-time, and uses an AI to formulate a context-aware response.

How It Works: You choose a single website to be the bot’s “source of truth” (e.g., a product page, a game’s status page, a live event schedule). When a user sends any message to the group, the bot instantly scrapes that website for the latest data, combines it with the user’s message, and sends it all to a powerful AI to get a relevant, intelligent answer.

Prerequisites:

Step 1: Configure the Data Source (ParseHub)

First, we need to teach ParseHub what information to grab from your target website.

  1. Create a New Project:

    • Log in to your ParseHub account.
    • Click “New Project” and enter the URL of the website you want the bot to use for information.

  2. Select Your Data:

    • Once the page loads, click on the first piece of text or data you want to extract.
    • Click a second, similar element to teach ParseHub to grab all matching items.
    • In the left sidebar, you will see a command like Select selection1. You must rename this to selection2 by clicking on the name. This is critical for the script to find the data.

  3. Get Your Project Token:

    • Click the gear icon next to “Project Settings” in the top-left menu.
    • Copy the Project Token. This is the unique ID for your scraping job.

Step 2: Set Up the Google Apps Script

This script is the complete brain of your bot.

  1. Go to script.google.com and create a “New project”.
  2. Delete the placeholder code and paste in the entire script below. This is the complete, ready-to-use code.
// --- Configuration ---
const SCRIPT_PROPERTIES = PropertiesService.getScriptProperties();
const PARSEHUB_API_KEY = SCRIPT_PROPERTIES.getProperty('PARSEHUB_API_KEY');
const PARSEHUB_PROJECT_TOKEN = SCRIPT_PROPERTIES.getProperty('PARSEHUB_PROJECT_TOKEN');
const GROUPME_BOT_ID = SCRIPT_PROPERTIES.getProperty('GROUPME_BOT_ID');
const GEMINI_API_KEY = SCRIPT_PROPERTIES.getProperty('GEMINI_API_KEY');
const RECENT_EXTRACTED_TEXT_KEY = 'RECENT_EXTRACTED_TEXT_HISTORY';
const HISTORY_LENGTH = 5; // How many past scraped data chunks to remember
// --- End Configuration ---

const PARSEHUB_API_BASE = 'https://www.parsehub.com/api/v2/';
const GROUPME_BOT_POST_URL = 'https://api.groupme.com/v3/bots/post';
const GEMINI_API_ENDPOINT = 'https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash-latest:generateContent';

/**
 * Handles all incoming POST requests from GroupMe.
 */
function doPost(e) {
  try {
    Logger.log('--- New Request Received ---');
    if (!PARSEHUB_API_KEY || !PARSEHUB_PROJECT_TOKEN || !GROUPME_BOT_ID) {
       Logger.log('FATAL ERROR: Script Properties are missing essential keys!');
       return ContentService.createTextOutput("Configuration Error.");
    }

    if (!e || !e.postData || !e.postData.contents) {
      Logger.log('WARNING: Received empty or invalid POST data. Ignoring.');
      return ContentService.createTextOutput("Received empty POST data.");
    }
    
    const requestData = JSON.parse(e.postData.contents);
    const messageText = (requestData.text || '').trim();
    const senderType = requestData.sender_type;

    // The bot responds to any message from a 'user' that is not empty.
    if (senderType === 'user' && messageText) {
      Logger.log(`Trigger condition met for user message: "${messageText}"`);
      sendGroupMeMessage("Thinking..."); // Let the user know the bot is working.

      const extractionResult = runParseHubAndExtractText();

      if (extractionResult.error) {
          Logger.log('ERROR during extraction phase. Sending error message to user.');
          sendGroupMeMessage(extractionResult.message);
      } else if (extractionResult.text) {
          const extractedText = extractionResult.text;
          Logger.log(`Successfully extracted text (length: ${extractedText.length}).`);
          
          updateExtractedTextHistory(extractedText, loadExtractedTextHistory());
          
          if (!GEMINI_API_KEY) {
              Logger.log('WARNING: GEMINI_API_KEY is missing. Cannot process AI response.');
              sendGroupMeMessage("I can see the data, but my AI brain is disconnected. (Admin: GEMINI_API_KEY is not set).");
              return ContentService.createTextOutput("Processed without AI.");
          }

          Logger.log('GEMINI_API_KEY found. Processing with AI.');
          const processedContent = processWithGemini(extractedText, messageText);
          
          if (processedContent) {
              sendGroupMeMessage(processedContent);
              Logger.log(`AI processing successful. Final message length: ${processedContent.length}`);
          } else {
              Logger.log('WARNING: AI processing failed or returned empty. Sending fallback message.');
              sendGroupMeMessage("I was able to get the data, but I'm having trouble thinking right now. Please try asking differently.");
          }
      } else {
           Logger.log("CRITICAL: Extraction returned no error but no text.");
           sendGroupMeMessage("Sorry, I couldn't retrieve valid content this time.");
      }
    } else {
      Logger.log('Message did not meet trigger conditions (was from bot or empty). Ignoring.');
    }
  } catch (error) {
    Logger.log('FATAL ERROR in doPost: ' + error.toString() + '\nStack: ' + error.stack);
  }
  Logger.log('--- Request Processing Finished ---');
  return ContentService.createTextOutput("");
}

/**
 * Runs the ParseHub project, waits for completion, and returns the data.
 */
function runParseHubAndExtractText() {
  Logger.log('Starting ParseHub scrape and extraction...');
  try {
    const startRunUrl = `${PARSEHUB_API_BASE}projects/${PARSEHUB_PROJECT_TOKEN}/run`;
    const startOptions = { method: 'post', payload: { api_key: PARSEHUB_API_KEY }, muteHttpExceptions: true };
    const startResponse = UrlFetchApp.fetch(startRunUrl, startOptions);
    if (startResponse.getResponseCode() >= 400) { Logger.log(`PH Start Fail: ${startResponse.getContentText()}`); return { error: true, message: `Error starting scrape (API Error)` }; }
    const runToken = JSON.parse(startResponse.getContentText()).run_token;
    if (!runToken) { Logger.log('PH Start Fail: No run_token in response.'); return { error: true, message: 'Error starting scrape (Invalid Response)' }; }
    Logger.log('ParseHub run started. Token: ' + runToken);

    const getRunUrl = `${PARSEHUB_API_BASE}runs/${runToken}?api_key=${PARSEHUB_API_KEY}`;
    let runStatus = ''; const maxWaitTime = 5 * 60 * 1000; const pollInterval = 15 * 1000; const startTime = new Date().getTime();
    Logger.log('Polling for run completion...');
    while (runStatus !== 'complete' && runStatus !== 'cancelled' && runStatus !== 'errored') {
      if (new Date().getTime() - startTime > maxWaitTime) { Logger.log('PH Timeout.'); return { error: true, message: 'Error: The website took too long to scrape.' }; }
      Utilities.sleep(pollInterval);
      const statusResponse = UrlFetchApp.fetch(getRunUrl, { method: 'get', muteHttpExceptions: true });
      if (statusResponse.getResponseCode() < 400) {
        runStatus = JSON.parse(statusResponse.getContentText()).status;
        Logger.log('Polling status: ' + runStatus);
      }
    }
    if (runStatus !== 'complete') { Logger.log(`PH Run Bad Status: ${runStatus}`); return { error: true, message: `Error: Scrape finished with status: ${runStatus}.` }; }
    Logger.log('Scrape complete. Fetching data...');
    
    const getDataUrl = `${PARSEHUB_API_BASE}runs/${runToken}/data?api_key=${PARSEHUB_API_KEY}&format=json`;
    const dataResponse = UrlFetchApp.fetch(getDataUrl, { method: 'get', muteHttpExceptions: true, headers: {'Accept-Encoding': 'identity'} });
    if (dataResponse.getResponseCode() >= 400) { Logger.log(`PH Fetch Fail: ${dataResponse.getContentText()}`); return { error: true, message: `Error fetching results (API Error)` }; }
    
    let decompressedDataText;
    try { const blob = dataResponse.getBlob(); decompressedDataText = Utilities.ungzip(blob).getDataAsString("UTF-8"); } catch (e) { decompressedDataText = dataResponse.getContentText("UTF-8"); }
    const scrapedData = JSON.parse(decompressedDataText);
    const extractedText = extractTextFromParseHubData(scrapedData);

    if (extractedText) {
        Logger.log('Extraction successful.');
        return { text: extractedText, error: false };
    } else {
        Logger.log('ERROR: Extraction failed, text was empty.');
        return { error: true, message: "I scraped the site, but couldn't find the content I was looking for." };
    }
  } catch (error) {
    Logger.log('FATAL ERROR during runParseHubAndExtractText: ' + error.toString());
    return { error: true, message: 'An internal error occurred during data retrieval.' };
  }
}

/**
 * Extracts combined text from the 'selection2' field in ParseHub data.
 */
function extractTextFromParseHubData(data) {
    let combinedText = "";
    try {
        if (data && Array.isArray(data.selection2)) {
            data.selection2.forEach(item => { if (item?.name) combinedText += String(item.name).trim() + " "; });
            return combinedText.trim() || null;
        }
    } catch(e) { Logger.log("Error during text extraction: " + e); }
    return null;
}

/**
 * Sends the scraped context and user message to the Gemini API for a response.
 */
function processWithGemini(contextText, userInstruction) {
   if (!GEMINI_API_KEY || !contextText || !userInstruction) return null;
   Logger.log('Constructing AI prompt...');

   //  ========== CUSTOMIZE YOUR AI's CORE INSTRUCTIONS HERE ==========
   const systemPrompt = `You are a helpful assistant. Use the following live data as your only context to answer the user's request. Be concise and directly answer the question based on the data provided.`;
   //  ================================================================

   const fullPrompt = `${systemPrompt}\n\n--- LIVE DATA CONTEXT ---\n"${contextText}"\n\n--- USER'S REQUEST ---\n"${userInstruction}"`;
   Logger.log('Full prompt being sent to AI (first 200 chars): ' + fullPrompt.substring(0,200) + '...');

   const payload = {
     contents: [{ parts: [{ text: fullPrompt }] }],
     generationConfig: { "temperature": 0.3 }
   };
   const options = { method: 'post', contentType: 'application/json', payload: JSON.stringify(payload), muteHttpExceptions: true };
   const apiUrl = `${GEMINI_API_ENDPOINT}?key=${GEMINI_API_KEY}`;
   try {
     Logger.log('Calling Gemini API...');
     const response = UrlFetchApp.fetch(apiUrl, options);
     const responseBody = response.getContentText();
     if (response.getResponseCode() >= 400) { Logger.log(`Gemini API Error: ${responseBody}`); return null; }
     Logger.log('Gemini API call successful.');
     const responseData = JSON.parse(responseBody);
     return responseData?.candidates?.[0]?.content?.parts?.[0]?.text.trim() || null;
   } catch (error) { Logger.log("FATAL ERROR calling Gemini API: " + error); return null; }
}

/**
 * Sends a message to the configured GroupMe bot.
 */
function sendGroupMeMessage(textToSend) {
    if (!GROUPME_BOT_ID || !textToSend) return;
    if (textToSend.length > 990) textToSend = textToSend.substring(0, 990) + "...";
    const payload = { bot_id: GROUPME_BOT_ID, text: textToSend };
    const options = { method: 'post', contentType: 'application/json', payload: JSON.stringify(payload), muteHttpExceptions: true };
    UrlFetchApp.fetch(GROUPME_BOT_POST_URL, options);
}

/**
 * Loads and updates the history of extracted text.
 */
function loadExtractedTextHistory() {
    try { const historyString = SCRIPT_PROPERTIES.getProperty(RECENT_EXTRACTED_TEXT_KEY); return historyString ? JSON.parse(historyString) : []; } catch (e) { return []; }
}
function updateExtractedTextHistory(newText, currentHistory) {
    currentHistory.unshift(newText);
    const updatedHistory = currentHistory.slice(0, HISTORY_LENGTH);
    SCRIPT_PROPERTIES.setProperty(RECENT_EXTRACTED_TEXT_KEY, JSON.stringify(updatedHistory));
}

Step 3: Securely Store API Keys & Tokens

Store your secret keys in Script Properties to keep them secure.

  1. In the Apps Script editor, click the gear icon (Project Settings) on the left.

  2. Scroll down to Script properties and click Add script property.

  3. Add the following four properties one by one, ensuring the names are an exact match (case-sensitive):

    • PARSEHUB_API_KEY (Get from your ParseHub account page)
    • PARSEHUB_PROJECT_TOKEN (The token from your project in Step 1)
    • GROUPME_BOT_ID (Your bot’s ID from the GroupMe developer site)
    • GEMINI_API_KEY (The API key you generated from Google AI Studio)

  4. Click Save script properties when you are done.

Step 4: Define the AI’s Persona (The System Prompt)

This is where you give the AI its core instructions. By changing one line of code, you can completely alter the bot’s personality and purpose.

Find the processWithGemini function in the script. The power to define your bot lies here:

//  ========== CUSTOMIZE YOUR AI's CORE INSTRUCTIONS HERE ==========
const systemPrompt = `You are a helpful assistant. Use the following live data as your only context to answer the user's request. Be concise and directly answer the question based on the data provided.`;
//  ================================================================

Examples to Spark Your Imagination:

  • For scraping product inventory:
    const systemPrompt = \You are an inventory check bot. The provided data is a list of products and their stock status. Answer the user’s question about product availability only.`;`

  • For scraping sports scores:
    const systemPrompt = \You are an excited sports commentator. Use the provided game data to give a high-energy, play-by-play style answer to the user’s question about the game.`;`

  • For scraping a list of events:
    const systemPrompt = \You are a helpful event coordinator. The context is a list of events and their times. Help the user figure out what is happening and when.`;`

Step 5: Deploy and Test

  1. Save Your Script: Click the Save project icon.
  2. Deploy as a Web App:
    • Click the blue Deploy button → New deployment.
    • Click the gear icon → Web app.
    • Set “Description” to something meaningful (e.g., “GroupMe AI Bot v1”).
    • Set “Who has access” to Anyone. This is required for GroupMe to reach your bot.
    • Click Deploy.
  3. Authorize Permissions when prompted and copy the Web app URL.
  4. Update Your GroupMe Bot: Go to the GroupMe Developer site, edit your bot, and paste the new Web app URL into the Callback URL field. Save your changes.
  5. Test It! Go to your GroupMe group and ask a question related to the content on your target website.

Understanding the Flow

When you ask your bot, “Is the blue shirt in stock?”:

  1. GroupMe sends your message to your script’s Web App URL.
  2. Your script sends “Thinking…” to the group.
  3. It triggers ParseHub to scrape your pre-configured clothing store page.
  4. The script waits for the scrape to finish and gets the data: “Red Shirt: In Stock, Blue Shirt: Out of Stock, Green Shirt: In Stock”.
  5. It builds a detailed prompt for the AI, combining your system prompt, the scraped data, and your question.
  6. The AI analyzes everything and generates a response: “The Blue Shirt is currently Out of Stock.”
  7. Your script posts this clear, helpful answer back to the group.

Troubleshooting with Logs

  • Bot Doesn’t Respond or is Very Slow: Remember, the bot scrapes a website every time someone talks, which can take up to a minute. If there’s no response:

    1. In your Apps Script project, click the Executions (stopwatch) icon on the left.
    2. Find the most recent failed or running execution of doPost.
    3. Click it to view the Logs. The logs will tell you exactly where the process stopped: a FATAL ERROR in configuration, a ParseHub timeout, or an AI API error. The logs are your most powerful debugging tool.

  • Bot Gives Strange Answers:

    • Check Scraped Data: Is ParseHub grabbing irrelevant ads or menus? Refine your selection2 in the ParseHub project to be more specific.
    • Refine Your System Prompt: Be more direct. Tell the AI exactly what to do and what to ignore.

If you have any suggestions or edits, please reply below.

And if you found this helpful, a :+1: would be much appreciated

5 Likes
4

Advanced: Set up a GroupMe bot that can read websites - General / GroupMe - JTech Forums

4 Likes