Model:

• LLM: OpenAI's GPT-4 [2] was provided with specialized knowledge using Retrieval-Augmented Generation (RAG), an approach that combines pre-trained LLMs with a real-time retrieval mechanism to enhance reasoning capabilities in unstructured settings.
• Embedding Process: Our local standard operating procedure (SOP) on CT protocols was indexed using OpenAI’s embedding model (text-embedding-ada-002-v2) [3]. Each sentence, alongside five preceding and five following sentences, was converted into a numerical embedding.

Approach:

• Cases: We created 100 synthetic cases of unstructured clinical requests, each containing a short patient history, the clinical question for imaging, as well as the requested imaging body region. 
• Query Matching: All cases were transformed into embeddings and matched via cosine similarity using LlamaIndex [4]. This approach enabled zero-shot learning for protocol retrieval.
• Retrieval-Augmented Generation: To ensure that the chatbot received targeted expert knowledge specifically aligned with the case description, the most relevant 15 text nodes were automatically retrieved from our local SOP for CT protocols for each clinical query.
• Reasoning process: The model was tasked to use a two-step reasoning process.
  • 1st Step: The chatbot identified the most relevant diagnosis, evaluated contrast agent contraindications, and matched protocols to anatomical regions.
  • 2nd Step: Using additional SOP context, the chatbot proposed the detailed CT protocol, contrast phases, and scan regions for each query (Fig. 1).

Evaluation:

• Performance rating by two radiologists as “correct,” “minor error,” or “wrong”
• Assessment on 100 synthetic cases