6

Recently I have seen a few new answers popping up for questions which are a few months or years old, the answers are not bad per-se but they feel like they were generated by an AI, e.g. Chat GPT.

stackoverflow reacted by banning such answers, but the problem of identifying them remains.

This is one example from SQA SE, I really don't know if it was written by a person or generated by an AI but it has the problem Chat GPT has- it "dances" around the point but don't properly answers it while keeping perfect language and structure.

Edit: here's an example

I copied the text of this question to ChatGPT and got a VERY similar answer:

When it comes to test automation for a chatbot, there are a few things that you should pay attention to:

Test Data Generation: Chatbots are designed to understand and respond to natural language input, which makes test data generation a critical aspect of chatbot testing. You'll need to create test data that includes a variety of scenarios, including different intents, entities, and user inputs.

RestAPI: RestAPI can be used to provide test data to your chatbot, but it may not be sufficient on its own. You'll need to develop a strategy for testing the chatbot's functionality and behavior as well.

Sandbox environments: It's often helpful to have access to sandbox environments for testing your chatbot. This will allow you to test your chatbot in an environment that mimics the production environment, but without the risk of affecting real users.

Guidelines: It's important to follow industry best practices when developing your chatbot test automation framework. Some key guidelines to consider include using version control, using continuous integration and delivery, and developing a robust test suite.

Differences from other test automation solutions: Chatbot testing can be more complex than other test automation solutions because chatbots are designed to handle natural language inputs. This requires specialized tools and techniques for generating test data and analyzing the chatbot's responses.

In terms of whether RestAPI is sufficient for testing your chatbot, it really depends on your specific use case. While RestAPI can be a useful tool for providing test data, it's likely that you'll need to incorporate other types of tests as well, such as functional tests that evaluate the chatbot's behavior in different scenarios.

4 Answers 4

Reset to default
5

Without an automated process to determine if an answer is written by an AI, any action we moderators take is going to be hit and miss.

I personally have taken to running any questions/answers that are flagged as possibly written by AI and checking them on https://writer.com/ai-content-detector/ Thus far almost every check I've made has come back with "mostly human".

There is a definite trend for people to flag longer answers that tend to be more general than specific as being AI-generated. This may or may not be the case.

My biggest issue with AI-generated answers is people using them without attribution. I see no problems with someone using something like a ChatGPT response as the basis of a posted answer, provided they both attribute the source and adjust the response to better fit the question they are answering. (Note: this is a personal opinion and not SO policy. As of the last I heard, each stack is free to choose how to handle AI-generated content)

4
  • 1
    Attribution is one thing, but the bigger problem I see is the creation of an infinite loops of AI generated answers based on other AI generated answers. Like you I have nothing against using AI (and not just ChatGPT) to look for answers as long as there is a human in the loop.
    – Rsf
    Commented Mar 13, 2023 at 9:14
  • +1 for mentioning attribution. When I see question about testing in context answered by James Bach I trust it because I know is has experience in that domain. When I google for "testing in context", I can decide about the credibility of the answer based on where it comes from. When reading Wikipedia article or blog post, I see a list of references (bibliography) to assess how credibile this work is. With AI generated answers I have no info, where the knowledge comes from (personal experience, peer-reviewed papers, etc.).
    – dzieciou
    Commented May 10, 2023 at 6:38
  • 4
    this user has been on a joy ride posting chatgpt answers (without attribution). I've seen enough chatgpt on stackoverflow to have a sense for it. compare their pre and post 2022-11-30 answers and you'll see a clear difference in writing style and answering rate.
    – starball
    Commented Jul 5, 2023 at 1:15
  • 2
    You don't need automated process. Humans are better AI detectors that tools. While it may be harder to detect AI posts for someone who hasn't seen enough of them I am pretty sure that there are Stack Overflow moderators that will lend a helping hand.
    – Dalija Prasnikar
    Commented Jul 5, 2023 at 14:25
1

How can we tell? Is it any different than reading several blog posts and combining them in one answer? It is actually written by a person, but you could say that for an AI generated answer - AI combined several posts written by humans in one unit.

0

I haven't been around here for a while, but I would strongly advocate for banning AI-generated content.

I can't help but noticing in this thread that at least two of the users I remember gaining lots of rep by posting lots of low-quality answers are not only still around, but now some of the prime suspects for doing the same but with AI-generated content.

That kind of behavior is a big part of why I got tired of contributing here a while back.

-2

So let's say an AI generated answer is posted on any question, if it helps resolve the problem, why shouldn't it be accepted? Also, how can we decide if an answer is AI generated or actually written by the person themselves?

Just like a person would use resources to learn any put things together, the AI too does the same. It analysis resources it has been trained on and accordingly give a response. Yes it may sometimes not be accurate, but the same can happen for a person writing the answer. A person may not give an accurate answer too.

5
  • 2
    you would say that though - your name is the only one on the front page... anything for internet points.
    – dvniel Mod
    Commented Feb 1, 2023 at 15:37
  • 1
    more like anything for helping the community.
    – IAmMilinPatel
    Commented Feb 3, 2023 at 8:44
  • 7
    Your answers aren't based on your experience, knowledge, or expertise though? You're basically copying someone else's homework in the hope that you'll get upvoted and accepted answers, and the way SE gamifies Q&A means you'll eventually kill the stack - people will stop asking and answering questions if ChatGPT's doing the work for both sides... then people will stop coming here altogether. You've gained 90 points in the last few days, purely from copying and pasting ChatGPT responses - extend that over the year, and that's ~32,000 points from zero effort.
    – dvniel Mod
    Commented Feb 3, 2023 at 8:59
  • 1
    You can verify the content here: writer.com/ai-content-detector. Yes, I use online services like Google, Blogs and even ChatGPT apart from getting help from friends and colleagues to try and find answers and solve problems. Based on the information gathered I put together an answer to problems I think I can help find solution to.
    – IAmMilinPatel
    Commented Jul 5, 2023 at 2:28
  • 3
    You are not gathering information, you are blatantly copy pasting from AI. Changing a word here and there can fool AI detection tools, but it will not fool people that have seen plenty of AI posts. We can easily spot the difference.
    – Dalija Prasnikar
    Commented Jul 5, 2023 at 14:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .