Vanta AI Case Study

“Generative AI is critical to Vanta’s roadmap across multiple products. We aim to deliver the best customer experience without compromising data privacy in the development process. Relari plays an instrumental role in our LLM product lifecycle, helping us systematically improve AI performance through rapid experimentation with custom synthetic datasets and high-quality metrics.”

– Tina Ding, Engineering Manager, Vanta AI and Enterprise Products 

‍

Who is Vanta?

Vanta is the leading automated trust management platform that helps companies achieve compliance with over 20 security and data privacy frameworks, including SOC 2, HIPAA, and ISO 27001. Vanta provides automated tests, workflows, and controls to ensure company compliance with these standards, allowing auditors to quickly complete audits and helping companies maintain compliance effortlessly.

As the market leader in the compliance technology space, Vanta has helped over 8,000 companies, including Atlassian, Omni Hotels, Quora, and ZoomInfo, build, maintain, and demonstrate their trust—all in a way that's real-time and transparent.

Vanta raised $150 million in their most recent Series C round, led by Sequoia Capital, at a $2.45 billion valuation (announcement).

‍

What brought Vanta to Relari

Vanta is an early adopter of Large Language Models (LLMs), integrating a suite of AI-driven tools into the Vanta platform to enhance various aspects of security and compliance. With Vanta AI, tasks that were previously impossible to automate can now be performed reliably in minutes, enabling security and compliance teams to prove trust and manage risk more efficiently and confidently than ever before. Some of the example use cases include:

• Questionnaire Automation: automatically drafts answers to security questionnaires, significantly reducing the time and effort required to provide accurate responses.
• Vendor Risk Management: a sophisticated chatbot that provides instant answers to security-related questions.
• Trust Center AI: Enhances the Trust Center by providing AI-driven insights and automated responses, ensuring that clients and prospects have access to accurate and timely information.

Additionally, Vanta laid out core principles and customer commitments that guide its AI product development (more details)

1. Do No Harm: Ensuring that AI use does not harm customers or Vanta by predicting and mitigating potential risks.
2. Security and Privacy by Design: Incorporating security and privacy from the start and with every change, supported by evidence.
3. Impact of Incorrectness: Assessing and managing the risks of errors.
4. Explainability and Transparency: Making AI results explainable and the processes transparent.
5. Data Control and Risk Management: Establishing control over data use and mitigating risks associated with data access and AI outputs.
6. Vanta does not train AI models on customers data. Vanta keeps customers in control of how data is used for AI systems and commits to leading by example on AI security and compliance.

With these use cases and principles in mind, Vanta’s AI team sought a way to systematically ensure the quality of each LLM-powered product without compromising data security. Relari.ai’s blog posts on data-driven LLM evaluation and development caught the attention of the AI/ML engineers, leading them to reach out to learn more about Relari.ai's approach.

‍

The Challenge: How to Systematically Improve LLM Applications in a Scalable and Data-Privacy Conscious Way

Vanta’s AI team faced several core challenges:

• Too much guesswork in building RAG. Most of Vanta’s AI products are powered by Retrieval-Augmented Generation (RAG), an architecture that uses a retriever to fetch relevant information from a knowledge database and then feeds it to the LLM to generate an answer. While it's simple to get started, selecting the optimal retrieval strategies and parameters is challenging. This turned the development process into a guessing game with hard-to-observe final results.

• Human evaluation is not scalable. Answers to security questions require a lot of judgment and are often difficult to evaluate. Although the team had experts to evaluate performance, it was unrealistic to test each change in the LLM pipeline and manually assess the quality of each answer and corresponding documents retrieved to diagnose issues.

• High quality standards from the outset. Poor AI products lose customer trust and good ones create trust. Vanta’s AI team has a very high standard for its LLM products and wants to ensure that the systems are robust even in alpha/beta versions. This challenges the team to find a way to productionize the AI product even prior to receiving feedback data from users.

‍

The Solution: Data-driven approach to systematically optimize LLM products

“A key component to building a robust RAG product is optimizing retrieval performance. Before we had Relari, we relied on guesswork and instincts to select key parameters such as similarity threshold, chunk size, embedding models, and retrieval strategies. Using Relari’s synthetic golden datasets and tailored evaluation metrics, we were able to easily understand trade-offs among different retrieval strategies, chunking logic, and parameters over a large dataset, and make confident, informed decisions. This data-driven process significantly improved our iteration speed, allowing us to quickly reach production-grade for multiple LLM products over a short period of time.”

– Noam Rubin, Vanta AI Software Engineer

‍

Leveraging the Relari.ai Data-Driven Toolkit, the Vanta team was able to tackle the above challenges using the following features:

• Tailored Synthetic Golden Datasets: Relari generates tailored synthetic golden datasets for each application, containing questions and the ideal (golden) expected outputs (retrieved documents and answers). The AI team runs the questions in the dataset through their system and sends the outputs to Relari’s API, which calculates granular metrics. With these datasets, Vanta’s team can stress test the application before launching to customers. (more detail on this approach).

• Custom Data-Optimized Metrics: Unlike naive LLM-as-a-judge metrics, Relari’s metrics combine deterministic and LLM-based methods based on rubrics provided by Vanta’s subject matter experts, capturing the unique requirements of each AI application. Relari then leverages human evaluation labels to optimize these metrics in a data-driven approach served through an API. (more details on this approach).

• Experimentation Dashboard: The data-driven experimentation workflow seamlessly integrates into Vanta’s internal software development process. This allows the team to easily visualize the performance and trade-offs of each experiment in Relari’s UI dashboard, drastically accelerating the development workflows.

Result: Successful deployments of AI products loved by customers

To date, Vanta has successfully launched multiple AI products that are now an integral part of their customers' trust management operations. These products have undergone rigorous testing, with over 50,000 evaluations conducted across six synthetic golden datasets, encompassing hundreds of diverse examples for each use case. This extensive testing ensures that the AI solutions are robust, reliable, and tailored to meet the specific needs of Vanta’s clients.

Vanta's new Questionnaire Automation product is just one example of how AI is making a real-world difference for their customers. By analyzing a customer's previously submitted security questionnaires — along with their existing security documentation and policies — Vanta AI is able to generate suggested responses to manual and time-consuming security questionnaires. This allows customers to simply review and approve, as opposed to starting from scratch each time. All told, Vanta's AI-powered Questionnaire Automation has helped customers like SmartRecruiters save 20 hours per week and enabled Noibu to complete security questionnaires 5x faster.

‍