Can AI Detect AI? A Multi Agent Framework for Identifying Large Language Models

Primary supervisor

Asad Malik

Co-supervisors

Large Language Models (LLMs) such as GPT, Llama, Qwen, and Mistral are increasingly used in commercial and academic applications. As more models become available, identifying which model generated a particular response becomes important for copyright auditing, model verification, and AI transparency.

Current fingerprinting methods often rely on manually selected benchmark questions. However, manually designing discriminative questions is time-consuming and may not capture unique behavioral differences between models.

This research proposes a multi-agent framework where several LLM agents collaborate to automatically generate and evaluate fingerprint questions for model identification.

Aim/outline

  1. Design a multi-agent system consisting of a question generator, evaluator, and refiner.
  2. Generate candidate fingerprint questions automatically.
  3. Evaluate how different LLMs respond to these questions.
  4. Measure the effectiveness of the generated questions for model identification.
  5. Compare multi-agent generated questions with manually selected questions.

URLs/references

References:

  • Instructional Fingerprinting of Large Language Models (NAACL 2024)
    https://aclanthology.org/2024.naacl-long.180/
  • TRAP: Targeted Random Adversarial Prompt Honeypot for Black-Box Identification (ACL Findings 2024)
    https://aclanthology.org/2024.findings-acl.683/
  • LLMmap: Fingerprinting for Large Language Models (USENIX Security 2025)
    https://www.usenix.org/conference/usenixsecurity25/presentation/pasquini
  • AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation
    https://arxiv.org/abs/2308.08155
  • CAMEL: Communicative Agents for Mind Exploration of Large Language Model Society
    https://arxiv.org/abs/2303.17760

Tools:

  • OpenAI API or local LLMs
  • Hugging Face Transformers
  • Sentence Transformers

Required knowledge

Must Have

  • Python Programming
  • Artificial Intelligence and Large Language Models (LLMs) Fundamentals
  • Basic Data Analysis
  • Research and Problem-Solving Skills

Recommended

  • Natural Language Processing (NLP)
  • Multi-Agent Systems
  • Hugging Face / OpenAI APIs
  • Prompt Engineering
  • Machine Learning Evaluation Metrics
  • Linux and Command Line Basics