AI Vendor Discussion Guide
Feel confident that you've checked all the boxes
🛫
🧠
🚣♀️
Getting started
Performance
Making it smart
Type of Tech: What's the core technology of your model? Is it open-source, wrapper around someone else’s, or entirely proprietary?
Deployment: If you manage deployment, how does it work? Can we deploy this on our private cloud, or migrate if/when we want to?
Infrastructure: What hardware and software requirements are needed for deployment?
Skills: What team skills are needed internally for successful deployment and maintenance? Do you offer training support and services?
Using Our Data: How can we integrate our proprietary datasets to enhance your model?
Data Retrieval: Can you describe the model's context window (how much it can intake at a time) and its maximum output length? How do you pull in more data when needed?
Who’s Involved: Do third-party tools process any of our data?
Knowledge Management: How can we integrate multiple knowledge bases with your model? How can we optimize efficiency when scaled?
🫶
Brand safety
Bias Prevention: What mechanisms are implemented to reduce bias and inappropriate outputs?
Toxicity Standards: Do you follow certain benchmarks for detecting harmful content? How frequently are these revised?
Output Filters: Are outputs subjected to any automated filters?
Content Restrictions: Can we implement guidelines to flag or block specific types of content?
🗣️
Feedback
Model Transparency: What tools are available to understand the model's decision-making process?
Insights: Are insights available in real time? How does the communication flow work in the event of a security issue?
Reporting: What kind of reports can be generated to gauge the efficiency, accuracy and performance of the model?
User Feedback Process: How do you manage and act upon user feedback?
🔐
Security
User Authentication: What authentication methods do you support for users?
Access Protocols: Who on your team can access the foundational models and our data? How long is data retained?
Single Sign-On (SSO): Do you facilitate SSO capabilities? Is SCIM supported (for example, OKTA)?
Prompt Security: How are measures implemented to counteract malicious prompts or exploitation events?
Handling Demand: How does your system cope during high-demand situations? How does performance fare with extensive datasets?
Dealing with Errors: How does the model handle “hallucinations”, i.e. erroneous outputs? Is there a human review process in place?
Quality Assurance: How do you measure your model's performance? How can we monitor the quality and accuracy of the model's outputs?
🧑⚖️
Compliance
IP Concerns: How do you ensure that generated content doesn't infringe on third-party IP rights? Are all outputs “enterprise safe”?
Data: Who retains ownership of the input data and generated content?
Privacy Standards: Which data privacy standards, like GDPR, do you adhere to?
Legal Track Record: Are there any past, ongoing, or anticipated legal issues related to your model?
Data Sources: From where did you source your training data? How did you limit bias during model training?
Third-party Evaluation: Has your model been evaluated by an external party? Do you have certifications related to bias, toxicity, or data security?