Asking it to find or tell me about our Supplier Management policy (or any other policy) is a poor use that interferes with individuals being accountable and responsible for learning their job. What happens in an audit, do you ask Siri to pull up the supplier management policy because you don't know the number or how to find it? How would one know if it is the right one if you hand over the responsibility for knowing your policies to Siri?
This is an interesting question.
If we ask: "What happens in an audit, do you ask Siri to multiply two 6-decimal place numbers because you don't know the multiplication? How would one know if it is the right answer if you hand over the responsibility for knowing your multiplication to Siri?"
In this example, we would commonly say that such digital calculators are accepted as ubiquitous tools in all walks and facets of life. In a ISO13485 (and perhaps others) ecosystem, we might answer that we have performed software validation and therefore have evidence that the answers are correct. [1]
If my QMS docs are stored in a website which lets me filter the view by word search and even between 'current revision only' and 'all revisions' of documents (e.g., show me the current version of all QMS docs with 'supplier' in the title) as a means to locate only the current document revision of the needed doc without needing to know some arcane QMS-PURCH-123-4567 doc number, does that interfere with individuals being accountable and responsible for learning their job? I would argue that it does not.
Similarly, couldn't we rely upon an LLM as a search tool, as much as we rely on our calculators, ERP, email, and other software and software services? If I ask for 'supplier management policy' and the LLM replies with the 'Corporate Policy on Office Supplies' QMS doc, I still have the responsibility for knowing that is not the correct document.
But I think your question raises a hugely important key point of the need to delineate clearly between where use of an LLM tool ends and the responsibility of the user begins. With something like a calculator or an ERP Purchase Order system, this is fairly obvious because of the very confined functionality of those software apps. But with an LLM, the nature of the LLM and manner of communication intentionally interjects its simulation into the realm we would usually reserve for human cognition and responsibility. Many of us have probably seen someone ask ChatGPT (or other LLM) a question and then take the response as unquestionably factual: they hand over their responsibility to discern and analyze to the LLM.
[1] This raises the question: is software validation of an LLM possible? Perhaps if we are careful to define the scope tightly, then it is. But the possible permutations of queries that users might submit, and the range of possible responses, seems indefinitely large.
