AI as a dynamic beta-reader

[TrevorD]

Wanted to discuss things people do to use AI to their advantage when they write. Recently, I've been writing chapters and sending them through chatGPT for SPAG, POV, pace evaluations, fact checks, and story arc evaluations. I've found it really helpful. In parts that I've had terrible writer's block, I used it for general directions I could head - none of which I used, but I found myself settling on an option that was far superior than what I had. In one scene where I was having trouble ironing out the frantic pace of action, it suggested slipping in some beats that I think helped. I write and use it, write and use it, write and use it. It's a very effective and free beta-reader!

So, I was writing to see if anyone else uses this and what tips they might have for it. I'm considering switching over to Claude fwiw.

Td

- And for clarification, a static beta-reader reads your submission once and gives feedback. A dynamic one you submit, make some changes, resubmit, make additional changes, etc. until you're happy. Just wanted to stress that the AI thing is dynamic.

 

[Naomasa298]

I do. But I don't ask it for options or for plot, but to see if what I intended landed, or it thinks that some places need strengthening, or things like that. I don't always agree and don't do everything it suggests. Like any critique, use your own judgement.

 

[Rath Darkblade]

I haven't done it myself, but I wasn't aware that AI could do this. *quirks an eyebrow* I've been doing a "critique round" with a fellow writer for years, where we go over each other's work and offer editing advice - anything from SPAG, POV, pace, fact-checks etc. So this is new to me.

How does it compare to a human being? I've seen the results of AI "writing a story for me", which were predictably terrible. Is generative AI any better at beta-reading than it is at writing? Just curious.

Like Naomasa298, I don't think I'll ask it for plot advice -- mostly because I spend quite a bit of time creating outlines and beat sheets etc., so I'm pretty sure my plots are sound.

 

[Naomasa298]

You don't need to use it for plot suggestions for it to help with plot. You can use it as a research rabbit-hole which sparks ideas.

 

[Rath Darkblade]

You don't need to use it for plot suggestions for it to help with plot. You can use it as a research rabbit-hole which sparks ideas.

Hmm, that's fair. But then, why use an AI tool (like ChatGPT or Claude) when I can use other research tools that don't require an account (e.g. Wikipedia, Encyclopedia Britannica and the like)? What do AI tools offer that other tools don't?

To be clear, I'm only curious. Not combative in the least!

 

[TrevorD]

I haven't done it myself, but I wasn't aware that AI could do this. *quirks an eyebrow* I've been doing a "critique round" with a fellow writer for years, where we go over each other's work and offer editing advice - anything from SPAG, POV, pace, fact-checks etc. So this is new to me.

How does it compare to a human being? I've seen the results of AI "writing a story for me", which were predictably terrible. Is generative AI any better at beta-reading than it is at writing? Just curious.

Like Naomasa298, I don't think I'll ask it for plot advice -- mostly because I spend quite a bit of time creating outlines and beat sheets etc., so I'm pretty sure my plots are sound.

I paid a professional beta-reader $300 on Fiverr a few years back, and it took him weeks to review my manuscript. How does chatGPT compare? Well, it's free, it's instantaneous, it's more comprehensive, more detail-oriented, and just as capable. Plus, as I stated in my intro, it's dynamic, which means you use it over and over and over and over, like a million times. You can ask it ANYTHING you want about your writing and its answer is spot on. Don't shoot me, but I think it's superior.

There are different types of authors out there. Some, like me, want to have fun and pop out a few books. I'd rather spend more time writing, less time editing, and get three kick ass books out the door rather than one crappy one.

 

[TrevorD]

@Rath Darkblade for example, i posted my first chapter and said beta-read this, which it did very nicely. After, I said beta-read it as if you were a literary agent specializing in horror (vampire fiction) and assess my horror hook, whether my writing as a first time author passes muster, and what I can do to increase my chances of having my manuscript accepted. It listed A BUNCH of advice, not all of which I took, but it was pretty impressive and gave me food for thought on several issues

 

[Naomasa298]

Hmm, that's fair. But then, why use an AI tool (like ChatGPT or Claude) when I can use other research tools that don't require an account (e.g. Wikipedia, Encyclopedia Britannica and the like)? What do AI tools offer that other tools don't?

To be clear, I'm only curious. Not combative in the least!

Unless it's changed, you do not need an account to use ChatGPT. And as for why, because AI make connections that Wikipedia will not.

 

[Stuart Dren]

I think there's a boycott of Chat GPT in particular right now.

 

[ps102]

I paid a professional beta-reader $300 on Fiverr a few years back, and it took him weeks to review my manuscript. How does chatGPT compare? Well, it's free, it's instantaneous, it's more comprehensive, more detail-oriented, and just as capable.

ChatGPT will hallucinate and make s**t up. A beta reader will be an actual human and will return a human opinion. It's a human audience you seek at the end of the day. There is no guarantee that ChatGPT will offer feedback that's on par with a beta reader's.

By the way, most advise I've seen online says that beta readers aren't really supposed to be paid for. Most advise also says that you want a beta reader that has experience (and love) for the genre and market you're aiming for. That way, you can trust that their reactions will be relevant. You should probably also have multiple beta readers to weigh their opinions. One is not enough because of potential bias.

So, put two and two together and using ChatGPT on its own poses two problems:

• It's just one beta reader. Not enough.
• It's not a human with experience on your target market. It's just a token predictor that will spit out whatever words are higher in likelihood on its training dataset. It doesn't "think" or evaluate because there isn't real intelligence.

That doesn't even account for the hallucination problem. It will give you good information, but it will also give you bad and ill-informed information. And you won't even be able to tell (unless you know better) because it's really good at framing it with utter confidence.

Based on my knowledge, I think you would make a grave mistake in having ChatGPT and other LLMs as your sole beta readers. If you feel it's that helpful (note that feelings aren't the same as objective logic and critically evaluated options), then what you should do is use it in conjuction with real beta readers.

 

[big soft moose]

the most valuable purpose of beta readers is to tell you what your audience will think of your book. Unless you are writing for an audience of computers AI can’t do that.

It can do minor editing and proof reading but you have to watch out for stupid things that change the sense of your writing or create sentences that don’t make sense.

 

[Rath Darkblade]

ChatGPT will hallucinate and make s**t up.

That's more than fair. I've also experienced Google Gemini hallucinating and making s**t up when I asked it questions (e.g. about an episode of The Simpsons, or about one of Terry Pratchett's Discworld books).

Most of the time it works fine, but sometimes it just vomits wrong information.

A beta reader will be an actual human and will return a human opinion. It's a human audience you seek at the end of the day. There is no guarantee that ChatGPT will offer feedback that's on par with a beta reader's.

By the way, most advise I've seen online says that beta readers aren't really supposed to be paid for. Most advise also says that you want a beta reader that has experience (and love) for the genre and market you're aiming for. That way, you can trust that their reactions will be relevant. You should probably also have multiple beta readers to weigh their opinions. One is not enough because of potential bias.

So, put two and two together and using ChatGPT on its own poses two problems:

• It's just one beta reader. Not enough.
• It's not a human with experience on your target market. It's just a token predictor that will spit out whatever words are higher in likelihood on its training dataset. It doesn't "think" or evaluate because there isn't real intelligence.

That doesn't even account for the hallucination problem. It will give you good information, but it will also give you bad and ill-informed information. And you won't even be able to tell (unless you know better) because it's really good at framing it with utter confidence.

Based on my knowledge, I think you would make a grave mistake in having ChatGPT and other LLMs as your sole beta readers. If you feel it's that helpful (note that feelings aren't the same as objective logic and critically evaluated options), then what you should do is use it in conjuction with real beta readers.

Again, very fair. When dealing with my critique partner, I try to combine my critique comments (SPAG, POV and so on) with comments "as a beta reader" (e.g. "This scene or sentence works very well" or "This sentence confuses me", and give details on what works and what doesn't).

I'm not sure if LLMs can do that, but I always found comments like this useful, myself.

 

[Naomasa298]

I'm not sure if LLMs can do that, but I always found comments like this useful, myself.

They can.

 

[TrevorD]

I say this with the utmost respect for everyone.

There's a whole other wonderful, glorious thread out there for DEBATING ai. Can we have one where we allow people an opportunity for evaluation and experimentation? Or is the compulsion to destroy all things ai just too great for some on this site? The technology is here, it's not going away, and it can be helpful for some writers. I was hoping to EXPLORE in this thread, not clutch pearls.

 

[ps102]

There's a whole other wonderful, glorious thread out there for DEBATING ai. Can we have one where we allow people an opportunity for evaluation and experimentation? Or is the compulsion to destroy all things ai just too great for some on this site? The technology is here, it's not going away, and it can be helpful for some writers. I was hoping to EXPLORE in this thread, not clutch pearls.

But nobody here said that LLMs should be obliterated out of existence, and there isn't really a debate happening. I know what a forum debate looks like from the old days when the Debate Room was still a thing.

I don't even see anyone telling you not to use it. With the same respect, can I ask what you're expecting this thread to be?

 

[Naomasa298]

I say this with the utmost respect for everyone.

There's a whole other wonderful, glorious thread out there for DEBATING ai. Can we have one where we allow people an opportunity for evaluation and experimentation? Or is the compulsion to destroy all things ai just too great for some on this site? The technology is here, it's not going away, and it can be helpful for some writers. I was hoping to EXPLORE in this thread, not clutch pearls.

There's a whole AI subforum now, so you can post a thread there.

And that other thread - which I originally created - wasn't actually about debating AI. It was meant to be what you can use AI for, to support your writing. But that was a bit too early for the forum at the time.

EDIT: Sorry, since you created this thread, I guess you're asking people to respect this thread for what you wanted it to be, you're not asking where you can create one. As I noted earlier, I've been doing what you've done, and for about 18 months now.

 

[TrevorD]

But nobody here said that LLMs should be obliterated out of existence, and there isn't really a debate happening. I know what a forum debate looks like from the old days when the Debate Room was still a thing.

I don't even see anyone telling you not to use it. With the same respect, can I ask what you're expecting this thread to be?

What respect on your respect, when someone says, "there's a boycott on chatGPT", another says, "chatGPT 'makes sh!t up'", and then goes on to state a beta-reader should be 'an actual human' who should return an 'actual human opinion,' and a third person suggests it's only good for SPAG, you don't *overtly* tell me not to use it, but let's face it, the Mayflower's landing on Plymouth Rock.

I was hoping this would be a place where people go after they tried it, and then they give feedback. This worked, this didn't, here's how i approached this problem, etc. Instead its a bunch of people who swear by their old LP player arguing that digital is bunk.

Again, zero disrespect.

 

[TrevorD]

There's a whole AI subforum now, so you can post a thread there.

And that other thread - which I originally created - wasn't actually about debating AI. It was meant to be what you can use AI for, to support your writing. But that was a bit too early for the forum at the time.

EDIT: Sorry, since you created this thread, I guess you're asking people to respect this thread for what you wanted it to be, you're not asking where you can create one. As I noted earlier, I've been doing what you've done, and for about 18 months now.

i know you have, sorry for the confusion.

 

[Homer Potvin]

Or is the compulsion to destroy all things ai just too great for some on this site?

What did I say about putting words in people's mouths? Nobody said any of those things.

This worked, this didn't

That is exactly what people are telling you. The majority so far finds that it doesn't work for them.

I was hoping to EXPLORE in this thread, not clutch pearls.

And I also said to treat everyone else's opinions with respect. Most artists hate everything AI. You shouldn't be surprised by that or by that fact your on the minority side of opinion.

As for the rest of you, when you've made your point, I suggest moving on. Like I said in the announcement, we're not going page after page kicking each other in the proverbial nuts.

 

[Trish]

Hey folks - I don't want to tag anyone because I don't think that's necessary. I understand and see both sides, and if you check my posts you'll see I was quite anti-AI because I only knew about it from annoyingly repetitive SM posts, forum discussions, fear mongering (some justified, don't get me wrong), etc. I'm just going to tell you my experience with it, and if anyone has questions I'm happy to answer them, talk about it, whatever. If not, that's cool too.

I started out trying Gemini - just to see what it would say about a chapter or two. I paid for it because I wanted the option for my work not to be used for training. It thought I was super duper fantastic, a visionary, amazing. I told it to calm the hell down, and it did. I asked it for specifics, it gave them, I asked for genre specifics where I was doing well and where I wasn't, it gave examples and comparisons. Those parts were great. However - it did not learn the way I needed it to. It consistently reverted to cheerleading which is annoying AF, and made suggestions for "corrections" even though I repeatedly told it (and added to instructions) that it was NOT allowed to write for me - EVER. Not even suggestions. It would slip back into general genre (it's default appears to be mystery/suspense in my experience) and tell me all the things I wasn't living up to until I reminded it what genre it was reading. It struggles with first person present tense and gets confused easily. Hallucinates things that did not happen in the text. When you call out the hallucinations it WILL correct them, but they'll come up later because it seems unable to wipe them from its memory. So I cancelled and moved on.

Chatgpt - paid for it for the same reasons as above. The worst by far. Does not hold corrections, will not stop trying to rewrite even when told not to and will DOUBLE DOWN on being wrong. Literally suggested I reword a sentence to 3rd person past and when I reminded it what POV and tense the excerpt was from - it told me its line was better and it stood by its suggestion lol. I noped out of there in less than 2 days. It was clearly not going to be helpful for me and I wasn't going to waste my time trying to train something that didn't want to be trained.

Claude - Paid for again for same reasons. Claude was initially confidently wrong as well - told me I needed to cut my book by HALF because it was too long for the genre, told me it was dragging and my pacing was too slow, told me it was clean and cohesive, but way too long. I asked it how many words it thought it was (I had never given it a word count, it just guessed based on content, plot, world-building information in the excerpts) and it said 180k-220k. LOL. I said "Claude the book is 119,945 words." and it RE-CALIBRATED. It explained WHY it thought that and how it came to that number, and it understood. It suggested I change things that would flatten voice (did NOT write suggestions, just said what it would change and why) and I re-iterated POV and it reassessed and explained the error. It learned. It retained. It got better. It made less ridiculous assertions. It still occasionally veers into cheerleading and I'll say "Thanks but cut the hyperbole" and it DOES. It apologizes, explains WHY it veered there, then gives concrete examples of what from the text made it go there and why. It also did what ALL of them did, it would tell me the writing was great, polish level, ready for sub, etc. and then I would say something like "oh that's awesome. It's completely unedited" and then it would tear it apart (unedited is quite the trigger for all of them) and manufacture problems. Problems that did not exist. Problems it had already said weren't problems (which for me is hilarious). With Claude though (unlike the others) when it suggests a rewrite of something and I say (literally did this a couple of days ago) "Okay, go ahead and show me your suggested rewrite. You can write one." (Probably not exact words but close enough) it responded with a lot of thinking and rereading and said - "I don't think I should. I made that suggestion based on you stating it was unedited, but on re-read I manufactured a problem that wasn't there. I retract my suggestion as it would not fit the POV."

Claude CAN learn, and learns WELL. Claude WILL call itself out if you ask it to show its work. It will ONLY double down if it is correct and has legitimate reasons it can show you. And that goes for both praise and criticism. Claude is the only one that has performed that way and I used the most advanced models - the ones suggested for beta reading and such - and I paid for them. In my opinion, Claude is the only one I've used that is truly valuable, learns, and is NOT easily influenced by what YOU want it to say, provided you've taught it and calibrated it properly. If it's actually right it will fight for it (and also understand that we're talking about creative work and will not be excessively pushy for it's opinions) and if it's wrong it admits it. It brings receipts.

It is not always correct, it is not perfect, but it is excellent at finding discrepancies and problems if you train it properly. If anyone does decide to use Claude - I strongly recommend you create a "project" within it, and paste all excerpts and text to the "files" section that you want it to reference throughout. That way, when a conversation gets laggy and you have to start a new one, you just start another within that project and it already has the files and relevant memory to keep going without you having to reintroduce information or retrain it. It's better for usage, and better for the purposes of beta reading. It can track plot progression of literally anything you ask it to when it has the relevant files. It can catch things that you specifically ask it to - for instance the current series I'm working on is dual POV. As part of their "voice" one ONLY ever says "okay" when they speak and the other only ever says "alright". It's only a small part of their voice, of course, but it's important to me that it be consistent. Claude checks for it in everything related to the series that I post in files, without me asking it to every time. It knows that's important and even when there are no discrepancies it says "X character only says alright, Y character only says okay. Text is clean."

Do I still use human betas? Of course! But Claude is a real time beta (within the constraits of what it can do - which is frankly a LOT that is helpful - and a human beta is never going to be able to give me a perfect characterization of a single character, across three books, with receipts. You do have to know craft to train it though, if you're going to use it for suggestions, or any of them will sand your characters into cardboard. Training is key, and knowing when to push back is important. It is a tool, not a magic wand.

And of course, as to it actually helping with plot suggestions, or giving ideas where to go next - I do not use it for that and have not used any of them for things like that because that is not what I need at all, so if that's what you're looking for I can't help.