Error in logic processing

Hi Upstage team,

I wanted to share a reasoning issue I found while benchmarking Solar Pro 4 Thinking. I think it is worth looking at because this was not simply a formatting mistake or a bad final transcription, the model made a genuine logical error in both its corrected code and its follow-up explanation.

I would also like to note that Ollama small q4 models (Gemma 4 31b, Qwen 27b) are able to solve this coding problem.

I also tried increasing the reasoning to MAX on Open Router and it kept making the same error.

The benchmark asks the model to debug a meeting-room scheduler containing exactly seven logical bugs. One of the rules is that every meeting requires exactly 15 minutes of cleanup afterward, meaning the conflict check must work symmetrically whether the new meeting comes before or after an existing booking.

The correct condition is:

not ( req["end"] + 15 <= booking["start"] or req["start"] >= booking["end"] + 15 ) 

Solar Pro 4 instead produced:

not ( req["end"] <= booking["start"] or req["start"] >= booking["end"] + 15 ) 

This is asymmetric. It gives the existing booking its required cleanup period, but does not give the new request its cleanup period when the new request occurs first.

That directly caused Solar to incorrectly place request R6 into Birch:

• R6 runs from 585–600.

• Birch already contains R2 starting at 600.

• Solar considered this valid because 600 <= 600.

• But R6 requires cleanup until 615, so R2 cannot start at 600.

• Birch is therefore unavailable, and R6 must instead be placed in Cedar.

I later asked Solar why it had put R6 in Birch. Rather than recognizing the mistake, it explicitly defended the result and said:

"no conflict with R2 (R2 starts at 600, after R6 ends at 600)."

That follow-up is important because it confirms that this was not an accidental output mistake. The model genuinely failed to apply the cleanup constraint symmetrically.

Most of the rest of the solution was strong: Solar correctly fixed the request ordering, capacity comparison, equipment comparison, deep-copy issue, room-selection ordering, and chronological output sorting. But this particular failure is significant because the benchmark specifically tests whether the model can maintain program-state constraints and correctly repair logical code.

For a new model in this size/capability class, I would consider this more serious than a minor benchmark miss. The rule itself is simple and explicitly stated in the prompt, yet the model:

1. failed to implement it correctly,

2. produced an incorrect downstream program state,

3. and then confidently defended the incorrect reasoning when challenged.

That combination, incorrect implementation plus confident persistence after the error is pointed out, is the part I think deserves attention.

I am excited to see a new contender like Sol Pro 4 in the medium sized class and will be happy to continue to test your model.

I hope this is useful for evaluation and debugging. I can also provide the full benchmark prompt if that would help reproduce the issue.