What role does a supplier play and how does that affect responsible AI?

When procuring AI-enabled products you might think that all vendors effectively do the same thing: build an AI system into a product. In our experience, there’s nuance here and it has effects later on in the responsible AI process. At its core, this boils down to what a vendor has access to. 

If a vendor is building an AI-driven chatbot, chances are they’re using a foundation model like OpenAI’s GPT or Claude by Anthropic. They connect to these kinds of AI models through an application programming interface (API) which acts as a kind of ‘direct-line’ to the service without the hassle of navigating through a graphical interface. That way, someone can build a chatbot for their site that’s GPT-powered without opening up ChatGPT. What this also means is that a vendor doesn’t have access to the actual AI model OpenAI or Anthrophic are operating. 

Why does not having access to a model make a difference?

1. After you choose a responsible AI framework, your supplier will need to test the AI system they want to sell to you.
2. Once they’ve tested the AI system, they would use the test results to ‘show’ you that their system actually complies with the responsible AI framework you chose.
3. Not having direct access to the model means that the test can only look at the information the model outputs rather than the system as a whole.

In simple terms, this is like having a maintenance check on a car but the mechanic only has access to the ignition, accelerator, brakes, and wipers. Of course, they would be able to see if the car drives well (if at all) and stops when it needs to, but subtle problems like a slow air leak due to a punctured tire would be hard to diagnose because the mechanic couldn’t check warning lights or tire pressure.

If, however, a supplier builds their own models, then they would have access to a model’s training data. Running in-depth tests for fairness issues and data quality on training data lets companies like Fairly spot biases and model issues that are ‘baked-into’ the AI system a supplier is selling. In addition, it allows a company looking to procure AI-driven products to have more insight into whether a given AI product is safe, reliable, and fair.

When a vendor limits access to the model they use, this can lead to them cherry-picking outputs from their model to make their model look more compliant than it actually is. In addition, a lack of model access means you won’t be sure that the model your supplier tests is the one they are providing to you. This might not even happen due to malicious intent, just as OpenAI updates its models in the background but lets programmers simply keep the “gpt-3.5-turbo” name in their code the same, similarly a supplier may provide test results for an AI system that, unknown to them, is simply out of date due to a model developer update.

Fairly's team is experienced in reviewing model documentation in industries as tightly-regulated as financial services. With the advent of AI models, we offers services to review AI model documentation as well. Connect with us so that we can help you better understand suppliers, models, and the AI landscape.

‍

DISCLAIMER. The insights and information contained in this article is intended for informational purposes only and we at Fairly AI do not intend for it to constitute legal advice, opinion, or recommendations that you, the reader, or others can depend on. For any legal issues linked to the information and insights contained in this article, please contact a lawyer/attorney to retain legal advice.

Fairly provides links to other websites beyond our control and we are not responsible for and do not give any warranties or make any representations regarding such websites. We are not responsible for or liable in relation to the content or security of these sites (other than to the extent required by law.)

Fairly makes reasonable endeavors to ensure reliability of the information presented on the app and website, but makes no warranties or representations whatsoever as to its accuracy or reliability. Similarly, due to the nature of the internet, we do not guarantee that the Fairly will be available on an uninterrupted and error free basis.

‍