Potential Regression in Python Analyzer Feedback Sensitivity (Post-Python 3.13.5 Upgrade)

Programming Python
1

Hi everyone,

I’m reaching out to report a potential issue with the automated feedback (Analyzer) on the Python track.

As a mentor, I’ve noticed a significant spike in the number of automated hints being triggered on solutions that previously passed flawlessly. Specifically:

  • Regressions on Old Solutions: Solutions I developed and passed months ago are now receiving “a ton of hints” upon resubmission.
  • Mentee Difficulty: Mentees are struggling to clear automated suggestions even with mentor oversight. In some cases, it’s unclear how to satisfy the current PyLint configuration without making the code less readable.

I noticed that the python-analyzer was recently updated to Python 3.13.5. I suspect that this upgrade, or the accompanying changes to the PyLint configuration, may have unintentionally enabled specific rules or increased the “noisiness” of the feedback.

Could the maintainers look into whether the recent environment update introduced unintended side effects in how the analyzer evaluates idiomatic Python code?

Examples of unexpected hints include:

Line 136 [C0104 disallowed-name] was reported by Pylint: Disallowed name "i.

Good work! :star2: Here are some general recommendations for improving your Python code.

Get familiar with the conventions outlined in PEP 8. While these are not "law", they are the standard used in the Python project itself and are a great baseline in most coding situations.
Read and think about the ideas outlined in PEP 20 (aka "The Zen of Python"). Like PEP 8, these are not "laws", but they are solid guiding principles for better and clearer Python code.
Prefer clear and easy to follow code over comments. But DO comment where needed for clarity.
Consider using type hints to clarify your code. Explore the type hint documentation and why you might not want to type hint.
Try to follow the docstring guidelines laid out in PEP 257. Good documentation matters.
Avoid magic numbers.
Prefer enumerate() over range(len()) in loops that need both an index and element.
Prefer comprehensions and generator expressions over loops that append to a data structure. But don't overuse comprehensions.
When joining more than few substrings or concatenating in a loop, prefer str.join() over other methods of string concatenation.
Get familiar with Python's rich set of built-in functions and the Standard Library. Go here for a brief tour and some interesting highlights.

Our Python Analyzer generated this feedback when analyzing your solution.

The second seems to show up for any solution not triggering something more specific.

Thanks for all the hard work on the track!

2

I believe these is intentional additional feedback from the analyzer.

That hint is a standard PyLint message,

3

Hi @stefanrothermel,

As a mentor, I’ve noticed a significant spike in the number of automated hints being triggered on solutions that previously passed flawlessly.

Previously, there were no Analyzer checks at all for practice exercises (the one exception being two-fer). There was no “passed flawlessly” - only “nothing to say, because nothing was checked.” The recent update changed that.

Regressions on Old Solutions: Solutions I developed and passed months ago are now receiving “a ton of hints” upon resubmission.

Anything changed or resubmitted is now being run against PyLint, so it will receive feedback. Depending on the solution, it may receive quite a lot of feedback.

Mentee Difficulty: Mentees are struggling to clear automated suggestions even with mentor oversight. In some cases, it’s unclear how to satisfy the current PyLint configuration without making the code less readable.

I don’t think the expectation should be to “clear” all the feedback unless the comment is flagged “essential” or “actionable”. But we may need to change summary comments to be clearer on that.

The point is to make the student aware of/think about places where their code might be better, or where it violates community standards like PEP8. But unless the UI is soft blocking a student from submitting the solution for publication, all Analyzer feedback is optional.

Examples of where PyLint suggestions make the code less readable would be greatly appreciated (code + the feedback + the changed code). AFAK, PyLint doesn’t have suggestions that push less readability for code …but if there are egregious examples, we can take a look at changing the text of the feedback or other mitigations.

The second seems to show up for any solution not triggering something more specific.

That second “hint” - AKA the general recommendations were developed in collaboration with a super-mentor, and are very general Python tips – no action really needed on the part of the student, just good genera things to think about.

If that is not clear from the celebratory intro message, then we’ll need to change it to be clearer. The tips are indeed given where there is no other specific feedback to give. They can be turned off/shortened, but we’d like to test them for a little bit before we do that.

1 Like
4

Hi all,

Thanks for the context. To help tune the new system, I’ve gathered some observations on how the shift to high-volume Pylint output is affecting the mentoring workflow and student behavior.

1. Signal vs. Noise (UX & Alert Fatigue) While the feedback is “optional,” students—especially diligent ones—treat automated hints as “tests to be passed.” When a specific PEP 8 fix is buried under 10 general tips and a celebratory intro, the actionable advice gets diluted.

Furthermore, the “Student received automated feedback” flag in the Mentor UI has lost its diagnostic value. Since it triggers on every submission now, I can no longer distinguish a student who is genuinely stuck from one who is just hitting “Submit.”

2. The “Seven-Attempt” Loop (Case Study) I recently watched a student submit seven consecutive iterations on the “Line Up” exercise. They successfully resolved a [C0103 invalid-name] hint by moving constant definitions to the module level, but the “General Recommendations” block continued to appear. Because this feedback didn’t change or disappear, the student interpreted it as a sign that their solution still wasn’t “right,” leading to seven unnecessary attempts to “fix” perfectly idiomatic code.

Because they interpreted the hint as a requirement, they kept guessing at names until they grew frustrated. This trial-and-error loop replaces learning with “appeasing the linter.”

3. Specific Configuration Issues Here’s one example where the current Pylint config pushes students away from idiomatic Python:

  • Disallowing _: Pylint flags the canonical “unused variable” name _ as a disallowed name. Students are replacing for _ in range(n) with dummy variables like for rng_idx in range(n), which is less idiomatic and more verbose.

Suggestions for “Tuning”:

  • Silence the “General Tips”: Could these be limited to the first 2-3 exercises of a track, or suppressed if the code meets a quality threshold?
  • Update .pylintrc: Allow _ as a variable name
  • Mentor UI: Perhaps “General Recommendations” shouldn’t trigger the “Automated Feedback” flag for mentors, keeping that flag reserved for specific, actionable issues.

I’m happy to keep gathering examples. My goal is to ensure these great new tools support the mentoring process rather than creating accidental friction for the students!

1 Like
5

P.S. To illustrate that this affects everyone: I just had a very experienced user (~5,000 reputation) request mentoring specifically because they were frustrated by the persistent “General Recommendations.” They wrote: “I tried many ways! However, now my code is still face automatically suggestions [sic]” followed by the PEP 8/PEP 20 tip block. If a power user thinks they are failing a check, a beginner has no chance of knowing the feedback is optional.

1 Like