Set up Data Cloud web crawlers to ingest and vectorize external pages for AI agents (opens in a new tab)
Readers will learn how to configure a Data Cloud Web Content Connector, filter target elements with CSS selectors, and generate searchable vector chunks from live websites.
Why read this
The guide also shows how to verify the ingested data using Data Explorer and SOQL.
The article introduces Retrieval-Augmented Generation and explains why keeping AI agents grounded in real-time data matters. It outlines the prerequisites of enabling Einstein, Agentforce, and Data Cloud in a development org.
It details the configuration of a Web Content Connector in Data Cloud Setup. The steps explain how to set crawl depth to zero to avoid following irrelevant links and how to use browser developer tools to copy CSS selectors for targeting specific page elements.
The guide describes creating a Data Lake Object and its corresponding Data Model Object. It notes that semantic search must be enabled while skipping image embedding and knowledge harmonization for this text-only use case. It concludes by showing how to validate the chunked output through Data Explorer and standard SOQL queries against the unstructured data model object suffix.
Read the original at Salesforce Break (opens in a new tab)