Action fails even if fail_on_error is false

[jeromecambon]

I'm giving the vale-action the following parameters:

• fail_on_error: false
• vale_flags: "--no-exit"

However, I still have the action to fail if reviewdog fails:

/bin/reviewdog -f=rdjsonl -name=vale -reporter=github-pr-check -fail-on-error=false -filter-mode=added -level=info
...
Error: reviewdog: Too many results (annotations) in diff.
  You may miss some annotations due to GitHub limitation for annotation created by logging command.
  Please check GitHub Actions log console to see all results.
...
Error: reviewdog exited with status code: 1

Whatever reviewdog is returning, Vale action should not fail if fail_on_error is false.

[jeromecambon]

Any workaround for this? Thanks!

[jeromecambon]

I have a fix for this, but don't have the rights to create a PR.

[michelle-purcell]

@jeromecambon - Thank you for raising this issue. I am also reproducing this problem too in a GitHub action.

The issue occurs regardless of whether I use actions/checkout@v3 or uses: actions/checkout@v2.

Here's an extract of the YAML configuration that reproduces the issue:

jobs:
  vale:
    name: Linting with Vale
    runs-on: ubuntu-latest
    permissions:
      actions: read
      checks: read
      contents: read
      pull-requests: read
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Vale Linter
        uses: errata-ai/vale-action@reviewdog
        with:
          fail_on_error: false
          vale_flags: "--no-exit --config=docs/.vale/vale.ini"
          filter_mode: diff_context
          files: docs/src/main/asciidoc/
        env:
          GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}

jdkato - Some context: We are implementing Vale as a pilot test in one project, and during the initial introduction of the Vale inter to the git workflow, we would like to ensure that a PR is not blocked, regardless of whether there are errors, warnings, or suggestions. Is there a workaround for this issue to prevent the build failure? Thank you.

[ElPaisano]

Hi all, we've recently been working on a vale gh action install in our docs as well, and were running into the same issue described here.

I may have found a temporary workaround:

Theory:
This error

Error: reviewdog: Too many results (annotations) in diff.
  You may miss some annotations due to GitHub limitation for annotation created by logging command.
  Please check GitHub Actions log console to see all results.

doesn't have anything to with vale or the vale config itself. Rather, in each of these cases, Vale is spitting out more annotations than the the allowed limit for GH actions, which is causing it fail, regardless of other configs like fail_on_error: false and vale_flags: "--no-exit".

Tests:
After looking over this thread, I made a few changes to our docs vale GH action config, partially inspired by @jeromecambon (Your file diff step was helpful, thank you!!!! 😄 ). The short summary of what I did was:

• Get the file diff first using jitterbit/get-changed-files@v1, and only have vale run on the diff, as opposed to the entire file and entire doc set. This is one way to limit the number of annotations.
• In the .yml file for the action, removed the --no-exit flag and reset fail_on_error to true because, in our case, we do want the action to fail if an ERROR is found
• Lastly, I told the vale GH action to only report issues of severity level ERROR and disregard WARNING and SUGGESTION (vale_flags: "--minAlertLevel=error"). In the contributor workflow I'm proposing, the contributor should have already run vale locally with a full config i.e. displaying ERROR, WARNING and SUGGESTION issues, and fixed those issues prior to commit and push. So, the GH action is just a gatekeeper at this point, only checking for the worst issues. This is another way to limit the number of annotations

Result:
Ran a few tests (spelling errors, grammar errors) and the action failed as expected. Removed the errors, and the action continued to work properly, and did not fail with the error described in this thread.

Conclusion:
In summary, the workaround was to limit the number of annotations to keep them below the GH action limit. So, depending on your workflow (run vale locally first with a full config, use the GH action as a gatekeeper for the worst offenders), this might sort of work as a solution

Further investigations:

• A GitHub App appears to allow an infinite amount of annotations, so gonna investigate that
• GH job summaries https://github.blog/2022-05-09-supercharging-github-actions-with-job-summaries/

[ElPaisano]

I'd be curious to see if this workaround works for other folks, and also hear @jdkato 's take on this.

PS I absolutely love vale, this is such a cool tool

[jeromecambon]

Thank you @ElPaisano for your update 😃
Not sure it would help me, since I get the issue on a single big file with many annotations.
Using jitterbit/get-changed-files@v1 one can only get a list of files, not the diff done by the PR. So I'm not sure to understand what you mean by "only have vale run on the diff, as opposed to the entire file"?
Vale is running on the entire file anyway, isn't it?

[jdkato]

@ElPaisano, thanks for looking into this.

I think you're right about this case: the Too many results error is a case in which the action probably should fail because the error shouldn't happen (unlike the action ignoring an error-level alert from Vale, which is a preference thing).

The question then becomes how do we avoid this, even for repos that have a lot of alerts to show? I don't have a ton of time currently to look into this, but some ideas I'd like to pursue are:

• Filter the results prior to running Vale (as we did in the previous version) instead of relying on reviewdog (Warning causes workflow to fail when fail_on_error is true #84 (comment)).
• Provide a better means of enabling only a subset of rules (Allow linting files with a single style or a subset of styles vale#466).
• At worse, only show an amount lower than the limit and display a warning message saying some alerts weren't displayed.

[jeromecambon]

@jdkato Any news on this one? Thanks!

[jdkato]

A new release should be out sometime next week.

[jdkato]

Sorry for the delay here -- I haven't had time to get the release finished yet. The good news is that a combination of changes (filtering order, moving away from Docker, etc.) have resulted in about a 50% improvement in performance.

[rmoff]

@jdkato is this something you're still planning to release a fix for? thanks!