RAI Toolkit
The following questions have been designed to help you assess risks throughout the implementation of AI projects. The questions are organized based on the AI product life cycle. We recommend you return to this tool throughout the project and complete the relevant questions. At the end you will be able to export answers as a PDF or JSON.
6. Integration & Deployment
6.1 Perform Operational Testing
- Has operational testing been performed to see whether the system works in an operational context, on the operational hardware?
6.2 Train Users
- What processes or training have been established to ensure operational end users understand system functionality, limitations, errors, flagging and reporting procedures, how to fix errors, etc.?
- Evaluate whether the training was successful.
- Establish additional measures or efforts to increase adoption and successful use.
6.3 Establish Incident Response Procedures
- Has a portal or system for flagging concerns been established?
- What protections exist for whistleblowers?
- If appropriate, create clear processes through which impacted stakeholders can request remedy or redress for negative impacts of the system.
- Is there an incident reporting and response process in place?
- Are system failures appropriately logged and flagged?
- Establish procedure through which root cause analyses will be conducted for system failures
6.4 Revisit Documentation and Security / Roll-up into Dashboards
- Establish auditing procedures or oversight mechanisms for:
- the overall system
- the components in the stack (sensors, data, model, infrastructure)
- operational users
- Set up dashboards/visualizations for senior-level tracking and review. What training and testing is done to ensure helpfulness and comprehension of the information?
- Do you have the required security and cybersecurity approvals to proceed?
6.5 Update Documentation
- Update SOCs, impact and risk assessments, CONOPS, security review, and data/model cards, and DAGR, as necessary.