A combination of factors. But primarily, the budget and William Shatner.
The basic premise of "the crew finds god, but god turns out to be an advanced computer/alien/energy being" is a classic one, but perhaps one that has been done too many times. And the story of "fanatics take over the ship and take it to Eden" was done in "The Way To Eden", which is notorious for being one of the worst episodes of the original series. So the thing wasn't off to a good start.
Then throw in the ILM, who had done most of the effects for the previous movies, wasn't available for the ST V. So they went to a cheaper effects house, whose effects proved to be sub-par. Some had to be redone, which led to even less budget elsewhere.
Then it was also too much "The Kirk Show". If you look at Star Trek IV, Nimoy was keen to give every character something to do. Everyone in the cast has at least one scene where they're doing something other than saying "Yes Captain!" Every actor has a moment to shine. But ST V is all about how great Kirk is. And note, what we get is toned down from what Shatner wanted. Remember how Spock and McCoy resisted Sybok's mind-whammy and stayed with Kirk? In the original script they betray him, like everyone else, and he has to save the day all by himself. The only reason that didn't happen is the Nimoy and Kelly point-blank refused to do the movie if Shatner did that to their characters.
Now you get into the details, and here the film just begins to look worse and worse...
Nimbus III is a good idea, but... digging holes in the desert as some sort of... what, farming? Huh? It's a baffling moment, and it's one of the establishing shots of the whole movie.
Pool played in an actual pool. Yeah, little background joke, not that important, but it's silly.
A cat lady stripper. Would be fine, if she was well executed. Today she'd be some CGI creation that looked great. But on this budget, just do a regular stripper. And Kirk defeats the cat lady by... throwing her into the 'pool'. Because cats hate water, see? <groan>
"Be one with the horse!" Some of the humour in this film is good, but some is just painful.
The Uhura fan dance. Maybe 25 years ago, Nichelle, but sorry, you're 57 years old and this just doesn't work. Apparently a writer suggested this as a joke, and Shatner immediately went with it. The writer didn't say anything because he wanted to be paid, but he could hardly believe that they'd gone with it.
Spock neck-pinching a horse. Nothing wrong with this as such, but it just... it's goofy, and it comes in the middle of what's meant to be an exciting action sequence.
Spock's rocket boots and the "Turbolift shaft of infamy". Ye gods, this is dumb, dumb, dumb. It may be the dumbest scene in any Star Trek movie. And honestly, I think it represents Shatner's directing style. Nimoy himself said that Bill isn't some bad guy, despite the various things he's done, but he's very big on rushing in without thinking things through or listening to advice, and he can be very bull-headed about it. I have to think there were people around him who could - and tried to - point out the obvious problems with the scene, and he just wasn't listening to them. A lot of the bad things in the film feel like that, actually. Like Shatner had an idea and just went with it, without too much thinking about how to make it work.
Sybok's mind-whammy is premised on the idea that
everybody hides some secret pain which is the primary driver of their actions, and that if they are shown this pain and made to openly confront it via mind rape, they will immediately give their undying loyalty to their rapist... because... um, reasons. Is it just straight mind-meld brainwashing? That could be interesting! By all means have Sybok be some Vulcan master, the most powerful telepath in Vulcan history, and one of Vulcan's greatest shames because of his behaviour. That's a potentially good take on what a rogue Vulcan could be, and why Vulcan's can't afford to have people doing that. But if he's controlling people then what's the "secret pain" about? And nobody ever says or even implies that Sybok is doing anything but genuinely showing people some repressed painful memory, so we're left to believe that this one thing in itself, simply remembering a bad memory that you didn't want to think about, is enough to over-write your entire personality. That's a major, even critical, element of the movie. And it's just plain ridiculous.
Getting to the centre of the galaxy in a few hours. No, sorry, but no. Not in Star Trek.
So there's this impenetrable barrier that's impossible to survive... then the ship just penetrated it anyway because... um, because they did. It's implied that they make it because they weren't afraid of it...? But then how did the Klingons get through? Was it because Klingons don't fear? But then why didn't Klingons get through ages ago? Incidentally the novelisation has Sybok creating some now form of shielding which gets them through, and the Klingons who are following scan it and copy it.
Scotty's head-bump. Again, a goofy comedy moment.
Uhura and Scotty suddenly in love with one another for no reason. A side effect of the conditioning? Who the hell knows. There was never one moment that indicated these two had feelings for one another previously, and there will never be one moment indicating that they had feelings for one another again. It just comes out of nowhere, serves no function, and then vanishes into history.
Kirk just had to refer to "Klingon bastards" again, doubtless because Shatner knew "You Klingon bastards, you've killed my son" was one of his better moments as Kirk, so why not just say it again in his movie? But stripped of the original context the line is left flat and awkward, and just serves to remind you of a better movie.
And on, and on.
Now I will say, Star Trek V is probably the Trek movie that best captures the friendship and camaraderie of the original series. With the camping trip and Sulu and Chekov lost in the woods and the banter between the crew and "you were never alone" and "not in front of the Klingons" (side note - when Kirk does die... he's not alone), there are nice touches here and there, and I applaud them. Oh, and it's a small thing but this film gave us the "
assault phaser", which is probably my all-time favourite phaser.
But frankly, even if this movie was a perfect version of itself, it wouldn't have been all that good. As it is, it's a very imperfect version of itself - and it's drowning under the mass of elements that are silly, goofy, and just don't make sense.