This article recaps a session by David Tyler and Jimmy Ledbetter, delivered during 3Cloud’s Envision Summit.
Once upon a time, Data Engineering was all about data architecture, fancy structures, and mind-boggling technical stacks. On the other hand, Knowledge Management was all about the thrill of classification, the art of search, and the triumph of retrieval. But hold your horses, because something magical happened when OpenAI joined forces with Microsoft’s Office 365 and Azure. Data Engineering and Knowledge Management fused into one glorious, unstoppable solution. This article discusses the impact OpenAI will have on our work and how it has effortlessly integrated into the very fabric of your everyday existence.
OpenAI has introduced transformative models that enable the processing of unstructured documents by utilizing text-trained models, encompassing data from sources like Wikipedia and Common Crawl. These models allow users to interact with documents, posing questions and receiving personalized responses, eliminating reliance on keyword searches.
The architectural foundation, including SQL databases and data lakes, remains, but enhancements enable the incorporation of unstructured files, such as PDFs or images. Notably, the introduction of a vector database involves converting documents into mathematical coordinates, facilitating interaction with AI in its mathematical language.
New layers, like the semantic kernel, permit automation and the generation of code bases. The data within models is frozen as of 2021, but plugins utilizing the semantic kernel can fetch up-to-date information from search engines, ensuring conversations involve the most current data.
The vector database finds application in scenarios like mergers and acquisitions, where unstructured data can be stored and interrogated directly. The semantic kernel, with its planner functionality, automates workflows, streamlining processes and achieving efficiencies.
Knowledge management and data engineering converge as data gatherers become integral in interrogating data, with generative AI assisting in creating requirements documents. This collaboration bridges the gap between creative business users and left-brained engineers.
While multiple technologies are at play, leveraging them is not mandatory for initiating generative AI projects. Starting can be as simple as setting up an OpenAI service and application, ensuring data protection and preventing unintentional model training on proprietary information.
AI provides a way to:
- Retrieve structure and unstructured assets the same way
- Interrogate and augment all assets to find insights or pattern
- Generate outputs for consumption in various formats or mediums
AI uses newfound powers like:
- Prompt Engineer typically, is more analytical and less technical with a strong understanding of business processes and problems.
- Vector Engineer typically, is more technical with a strong understanding of data engineering space. It works closely with businesses to identify indices to build.
- Document Engineer typically, is more business focused and less technical and can be used to identify documents to ingest. It uses different cognitive services to ensure documents are ready for ingestion.
The Future of AI
In the future, businesses will transition from single intelligent agent scenarios to multi-intelligent agents using multi-modal Generative AI solutions. Low-risk repetitive tasks will be automated entirely using multi-modal Generative AI solutions. Generative AI will create, action, and manage experimentation of new market segment development.
AI and 3Cloud
See how 3Cloud has guided our clients in using of the power of Artificial Intelligence by learning more about our experience with Forsyth County Schools and a large tech company. Contact us to learn more about how AI can impact your business. Let’s get started!