Generated Caption Quality Tuning

Problem

Transcript quality tuning needed evidence before threshold changes. The real-video export showed many useful knowledge-dense videos landing as usable, and one visual-heavy case landing as weak.

The important question was whether the thresholds were too strict or whether one signal was being misrepresented.

Context

Summry scores transcript quality with deterministic penalties: segment count, segment length, filler density, visual references, repeated segments, sentence completion, generated captions, and language confidence.

Generated YouTube captions deserve some caution, but generated does not mean the transcript language is wrong. Treating generated captions as zero language confidence stacks two different penalties for one piece of evidence.

Implementation

I kept the transcript quality thresholds unchanged.

Instead, I changed YouTube transcript fetching so generated captions keep is_generated = true but report language_confidence = null. That means the scorer still applies the small generated-caption caution, but it no longer applies the separate low-language-confidence penalty unless a real confidence score exists.

I also added tests for the distinction:

• generated caption fragments without explicit low language confidence can still score strong
• explicit low language confidence still lowers the score to usable
• generated YouTube transcript payloads now leave language confidence unknown instead of falsely low

Tradeoffs

This is narrower than changing thresholds. It avoids weakening the quality gate broadly while fixing the specific double-penalty that made generated captions look worse than the evidence supported.

The tradeoff is that some generated-caption videos may now score stronger than before. That is acceptable because generated captions still carry their own warning, and later summarization prompts already include caution behavior when eligibility requires it.

Lessons Learned

Threshold tuning should start by validating inputs. A bad signal can look like a strict threshold problem, but changing thresholds would spread the fix across every transcript source.

In this case, generated-caption status and language-confidence status needed to stay separate. One describes how the transcript was produced; the other should describe confidence in language detection.