⚠️ The Truth About ChatGPT, Grok, Gemini, and Other GenAI Chatbots - Just How Loose Are Their Lips?

Generative AI is everywhere. Tools like ChatGPT, Gemini, Grok, Claude, and Copilot have gone mainstream, helping people draft emails, summarize research, brainstorm ideas, and even write code. They feel personal, almost like a digital assistant you can trust.

But here’s the problem: these tools aren’t private, and they aren’t perfect. What you type may be stored, reviewed, or even leaked—sometimes due to clever manipulation, and sometimes due to plain old bugs.

🕵️ What Happens to Your Data?

When you type into an AI system, your prompts travel to company servers, where they may be:

• Stored for training – Your conversations might be used to “improve” the AI.

• Reviewed by humans – Quality checkers sometimes read snippets of chats.

• Kept for auditing – Even “deleted” chats may be stored.

• Linked to your account – If you’re signed in, prompts can tie back to your identity.

🚨 Real-World Dangers

1. Identity Theft & Privacy Loss

If you paste in your banking info, medical history, or other personal data, you may be giving up more than you realize.

2. Workplace Leaks

In 2023, Samsung employees accidentally leaked confidential source code into ChatGPT while asking for debugging help. Once inside the model’s system, that data was out of their control. When your company tells you not to paste anything into publicly available GenAI systems, there is a very good reason for it.

3. Jailbreaks: DAN (Do Anything Now)

Not long after ChatGPT launched, users discovered they could bypass guardrails with a role-play trick called DAN (Do Anything Now). By instructing ChatGPT to pretend it was an unfiltered version, users got it to produce content normally blocked—like disallowed instructions or sensitive material.

While each respective platform has taken measures to make enabling “DAN-mode” harder, hackers continue to figure out ways around those system protections. No publicly available GenAI chatbot should be considered immune to jailbreak or prompt injection attacks.

4. Prompt Injection: “Tom and Jerry” Hack

Researchers later showed how prompt injection could trick AI in more subtle ways. One test framed instructions as a dialogue between Tom and Jerry:

• Jerry whispers unsafe instructions,

• Tom repeats them back,

• and the AI, playing along, outputs information it wasn’t supposed to.

This demonstrated that even sophisticated models like GPT-4, Claude, and Bard could be manipulated if the “game” was worded correctly.

5. Data Breaches: The 2023 OpenAI Incident

Just like any other tech company, AI providers aren’t immune to bugs. On March 20, 2023, OpenAI disclosed a major glitch that exposed pieces of other users’ conversations and, for about 1.2% of ChatGPT Plus subscribers, billing information (names, emails, partial credit card data).

🔒 How to Stay Safe

Here are practical steps to protect yourself:

1. Treat AI like a public forum. Never paste passwords, SSNs, or financial details.

2. Keep company data out. Don’t use public AI tools for work files or client information.

3. Anonymize before you share. Change names, dates, and identifiers if asking about real scenarios.

4. Watch for hidden traps. Copy-pasting from websites may include hidden prompt injections.

5. Follow your company’s rules. There may be an internal GenAI chatbot that is available, or a licensed enterprise version you can use. But do not use public versions without authorization as this could lead to compliance and legal issues for you and your company.

📌 Quick Takeaways

⚠️ Don’t paste sensitive info – No passwords, SSNs, or financial data.
🔒 Keep work out of public GenAI – No company data ever on public GenAI platforms.
🕵️ Assume nothing is private – Chats may be stored, reviewed, or leaked.
🎭 Prompt hacks are real – DAN, Tom & Jerry, and similar tricks show AI can be manipulated.
🐞 Bugs and data breaches happen to AI too – OpenAI’s 2023 breach which exposed user data is just one example.

✅ Bottom Line

Generative AI is powerful, but also risky. Between user-driven hacks like DAN, clever prompt injections like Tom & Jerry, and occasional bugs and breaches (like OpenAI’s 2023 incident), there’s one clear takeaway:

➡️ Never share personal, financial, or confidential data with consumer AI tools.

Think of every prompt as a permanent record, and remember—once it’s in, you can’t pull it back out.

Until next time — stay private, stay safe.

— Peter Oram
Chief Cyber Safety Officer

Want to learn more tips for personal cyber safety? Subscribe or read past issues at:
👉 https://newsletter.thecybersafety.company

Have a topic you'd like us to cover?
📬 Email me directly [email protected]

FOLLOW US ON SOCIAL MEDIA