https://www.bcs.org/articles-opinion-and-research/generative-ai-observability-issues-and-solutions/ ù
Understanding the issues
For the remainder of the article, the example I’ll use is Gen AI delivered as a black box REST API by a vendor. In preparation for this article, I listed all the standard business, operational, and security risks of incorporating a vendor service into your software product. I then listed the new risks that attend a product like ChatGPT, Bard, or Claude. I was shocked at the result.
The category of new technology risks that are not visible by automated means is:
- Hallucination
- Lying
- Alignment
- Unfairness, bias, toxicity
- Catastrophic forgetting
- Model collapse
- Model poisoning
- Drift (data, model, concept)
While the list of visible technology risks is much shorter:
- Semantic drift
- Factual inaccuracy
- Sentiment drift