Stabilizing asynchronous login failure processing by ahus1 · Pull Request #52503 · keycloak/keycloak

ahus1 · 2026-09-08T08:30:37Z

Root cause

Same as #52428 / #52429: DefaultBlockingBruteForceProtector processes failure counting asynchronously via an executor. When two login failures for the same user arrive in quick succession, the second request hits isLoginInProgress() == true, gets a forceChallenge() instead of failureChallenge(), and failedLogin() is never called — so the failure counter is not incremented. The counter stays at 1 instead of reaching failureFactor=2, and subsequent lockout assertions fail.

PR #52429 fixed this for AttackDetectionResourceTest (new testsuite, password grant). The same race exists in BruteForceTest and LDAPReadOnlyTest (old testsuite), both for browser-based and password-grant login attempts.

Approach

Added WaitUtils.waitForBruteForceExecutors(testingClient) after each invalid login attempt to ensure the async executor task completes before the next request is sent. This is the old-testsuite equivalent of the awaitNumFailures() polling used in PR #52429 for the new testsuite.

The wait is added inside the shared helper methods (loginInvalidPassword(), sendInvalidPasswordPasswordGrant()) so all callers are covered, plus explicit waits at direct getTestToken() call sites. Redundant standalone waitForBruteForceExecutors calls that existed before are removed.

Once #52128 makes brute force processing synchronous, these waits become unnecessary.

Closes keycloak#52502 Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>

keycloak-github-bot

Unreported flaky test detected, please review

keycloak-github-bot · 2026-09-08T09:52:11Z

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.forms.MultipleTabsLoginTest#multipleTabsLoginAndPassiveCheck

Keycloak CI - Forms IT (chrome)

org.opentest4j.MultipleFailuresError: 
Multiple Failures (2 failures)
	org.openqa.selenium.TimeoutException: timeout: Timed out receiving message from renderer: 10.000
  (Session info: chrome=152.0.7977.64)
Build info: version: '4.46.0', revision: 'df5a634 df5a6341cbb5f8f06b836bd595752775627c80dc'
...

Report flaky test

org.keycloak.testsuite.forms.MultipleTabsLoginTest#multipleTabsParallelLoginTestWithAuthSessionExpiredInTheMiddle

Keycloak CI - Forms IT (chrome)

org.openqa.selenium.TimeoutException: 
timeout: Timed out receiving message from renderer: 10.000
  (Session info: chrome=152.0.7977.64)
Build info: version: '4.46.0', revision: 'df5a634 df5a6341cbb5f8f06b836bd595752775627c80dc'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '6.17.0-1022-azure', java.version: '25.0.4.1'
...

Report flaky test

org.keycloak.testsuite.forms.MultipleTabsLoginTest#loginActionWithoutExecutionInRequiredActions

Keycloak CI - Forms IT (chrome)

org.openqa.selenium.TimeoutException: 
timeout: Timed out receiving message from renderer: 10.000
  (Session info: chrome=152.0.7977.64)
Build info: version: '4.46.0', revision: 'df5a634 df5a6341cbb5f8f06b836bd595752775627c80dc'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '6.17.0-1022-azure', java.version: '25.0.4.1'
...

Report flaky test

org.keycloak.testsuite.forms.MultipleTabsLoginTest#multipleTabsParallelLoginTestWithAuthSessionExpiredAndRefreshInTab1

Keycloak CI - Forms IT (chrome)

org.openqa.selenium.TimeoutException: 
timeout: Timed out receiving message from renderer: 10.000
  (Session info: chrome=152.0.7977.64)
Build info: version: '4.46.0', revision: 'df5a634 df5a6341cbb5f8f06b836bd595752775627c80dc'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '6.17.0-1022-azure', java.version: '25.0.4.1'
...

Report flaky test

org.keycloak.testsuite.forms.MultipleTabsLoginTest#testLoginPageRefresh

Keycloak CI - Forms IT (chrome)

org.openqa.selenium.TimeoutException: 
timeout: Timed out receiving message from renderer: 10.000
  (Session info: chrome=152.0.7977.64)
Build info: version: '4.46.0', revision: 'df5a634 df5a6341cbb5f8f06b836bd595752775627c80dc'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '6.17.0-1022-azure', java.version: '25.0.4.1'
...

Report flaky test

org.keycloak.testsuite.forms.MultipleTabsLoginTest#loginWithDifferentClients

Keycloak CI - Forms IT (chrome)

org.openqa.selenium.TimeoutException: 
timeout: Timed out receiving message from renderer: 10.000
  (Session info: chrome=152.0.7977.64)
Build info: version: '4.46.0', revision: 'df5a634 df5a6341cbb5f8f06b836bd595752775627c80dc'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '6.17.0-1022-azure', java.version: '25.0.4.1'
...

Report flaky test

org.keycloak.testsuite.forms.MultipleTabsLoginTest#multipleTabsParallelLoginTestWithAuthSessionExpiredAndRegisterClick

Keycloak CI - Forms IT (chrome)

org.openqa.selenium.TimeoutException: 
timeout: Timed out receiving message from renderer: 10.000
  (Session info: chrome=152.0.7977.64)
Build info: version: '4.46.0', revision: 'df5a634 df5a6341cbb5f8f06b836bd595752775627c80dc'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '6.17.0-1022-azure', java.version: '25.0.4.1'
...

Report flaky test

org.keycloak.testsuite.forms.MultipleTabsLoginTest#multipleTabsParallelLoginTest

Keycloak CI - Forms IT (chrome)

org.openqa.selenium.TimeoutException: 
timeout: Timed out receiving message from renderer: 10.000
  (Session info: chrome=152.0.7977.64)
Build info: version: '4.46.0', revision: 'df5a634 df5a6341cbb5f8f06b836bd595752775627c80dc'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '6.17.0-1022-azure', java.version: '25.0.4.1'
...

Report flaky test

org.keycloak.testsuite.forms.MultipleTabsLoginTest#expiredAuthenticationAction_expiredCodeCurrentExecution

Keycloak CI - Forms IT (chrome)

org.openqa.selenium.TimeoutException: 
timeout: Timed out receiving message from renderer: 10.000
  (Session info: chrome=152.0.7977.64)
Build info: version: '4.46.0', revision: 'df5a634 df5a6341cbb5f8f06b836bd595752775627c80dc'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '6.17.0-1022-azure', java.version: '25.0.4.1'
...

Report flaky test

org.keycloak.testsuite.forms.MultipleTabsLoginTest#testInjectRedirectUriInClientDataAfterAuthSessionExpiration

Keycloak CI - Forms IT (chrome)

org.openqa.selenium.TimeoutException: 
timeout: Timed out receiving message from renderer: 10.000
  (Session info: chrome=152.0.7977.64)
Build info: version: '4.46.0', revision: 'df5a634 df5a6341cbb5f8f06b836bd595752775627c80dc'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '6.17.0-1022-azure', java.version: '25.0.4.1'
...

Report flaky test

Copilot

🟡 Changes recommended

One or more issues must be addressed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Stabilizes legacy Arquillian brute-force tests by waiting for asynchronous login processing.

Changes:

Adds waits between repeated invalid login attempts.
Centralizes waits in shared login helpers.
Documents the wait utility and shortens its polling interval.

File summaries

File	Description
testsuite/integration-arquillian/tests/base/src/test/java/org/keycloak/testsuite/forms/BruteForceTest.java	Updated as part of this pull request.
testsuite/integration-arquillian/tests/base/src/test/java/org/keycloak/testsuite/federation/ldap/LDAPReadOnlyTest.java	Updated as part of this pull request.
testsuite/integration-arquillian/tests/base/src/main/java/org/keycloak/testsuite/util/WaitUtils.java	Updated as part of this pull request.

Review details

Suppressed comments (5)

testsuite/integration-arquillian/tests/base/src/test/java/org/keycloak/testsuite/forms/BruteForceTest.java:320

The first invalid OTP grant can still race with the asynchronous successful grant at line 310, which uses the same executor and loginAttempts tracking. Wait for that successful setup task before sending the request at line 316.

            WaitUtils.waitForBruteForceExecutors(testingClient);

testsuite/integration-arquillian/tests/base/src/test/java/org/keycloak/testsuite/forms/BruteForceTest.java:370

The first missing-OTP grant can still race with the asynchronous successful grant at line 360, which keeps the same loginAttempts entry until its executor task commits. Add the barrier after the successful setup request too.

            WaitUtils.waitForBruteForceExecutors(testingClient);

testsuite/integration-arquillian/tests/base/src/test/java/org/keycloak/testsuite/forms/BruteForceTest.java:422

The first loop iteration is not serialized with the successful grant at line 412. Since successful logins are processed by this executor too, add a wait after that setup request or the first OTP failure can be force-challenged without incrementing the counter.

                WaitUtils.waitForBruteForceExecutors(testingClient);

testsuite/integration-arquillian/tests/base/src/test/java/org/keycloak/testsuite/forms/BruteForceTest.java:1452

lockUserWithPasswordGrant() sends its first invalid request immediately after a successful grant, but successful logins also occupy loginAttempts until their asynchronous executor task commits. Drain the executor after that setup grant so the first loop iteration cannot be skipped.

        WaitUtils.waitForBruteForceExecutors(testingClient);

testsuite/integration-arquillian/tests/base/src/test/java/org/keycloak/testsuite/forms/BruteForceTest.java:1164

The first wrongpass request is still unsynchronized with the successful browser login above. That success also runs asynchronously and can keep loginAttempts occupied, so wait after expectLoginSuccess(...) before sending the first failure.

        WaitUtils.waitForBruteForceExecutors(testingClient);

Files reviewed: 3/3 changed files
Comments generated: 4
Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

        String sessionId = EventAssertion.expectLoginSuccess(events.poll()).getEvent().getSessionId();

        getTestToken("wrongpass", totpSecret);
+        WaitUtils.waitForBruteForceExecutors(testingClient);


Copilot

🔵 Needs a closer look

One or more issues must be addressed before approval.

Review details

Suppressed comments (1)

testsuite/integration-arquillian/tests/base/src/test/java/org/keycloak/testsuite/forms/BruteForceTest.java:1052

This drains the failed request, but the successful token request below also clears failures asynchronously and is followed immediately by the numFailures == 0 assertion. Drain the executor after that successful response too, otherwise this test can still observe the old count of 1.

        WaitUtils.waitForBruteForceExecutors(testingClient);

Files reviewed: 3/3 changed files
Comments generated: 0 new
Review effort level: Balanced

Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>

Copilot

🟢 Approval recommended

No unresolved issues were identified that would block approval.

Review details

Files reviewed: 3/3 changed files
Comments generated: 0 new
Review effort level: Balanced

Copilot

🔵 Needs a closer look

One or more issues must be addressed before approval.

Review details

Suppressed comments (1)

testsuite/integration-arquillian/tests/base/src/test/java/org/keycloak/testsuite/forms/BruteForceTest.java:1403

Adding the barrier only to the password helper leaves the same race in the browser OTP paths: loginWithTotpFailure() and repeated continueLoginWithInvalidTotp() calls enqueue OTP failures without waiting, while OTPFormAuthenticator checks brute-force state before each attempt. An in-flight OTP failure can therefore turn the next attempt into a force challenge and skip its counter increment; add the wait to those failure helpers too.

        WaitUtils.waitForBruteForceExecutors(testingClient);

Files reviewed: 3/3 changed files
Comments generated: 0 new
Review effort level: Balanced

Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>

Copilot

🟢 Approval recommended

No unresolved issues were identified that would block approval.

Review details

Files reviewed: 3/3 changed files
Comments generated: 0 new
Review effort level: Balanced

ahus1 self-assigned this Sep 8, 2026

keycloak-github-bot Bot added the team/continuous-delivery label Sep 8, 2026

ahus1 force-pushed the is-52498-stabilize-bf-test branch from 000c053 to ebd5280 Compare September 8, 2026 08:39

Stabilizing asynchronous login failure processing

b08a6ec

Closes keycloak#52502 Signed-off-by: Alexander Schwartz <alexander.schwartz@ibm.com>

ahus1 force-pushed the is-52498-stabilize-bf-test branch from ebd5280 to b08a6ec Compare September 8, 2026 08:41

keycloak-github-bot Bot added the flaky-test label Sep 8, 2026

keycloak-github-bot Bot reviewed Sep 8, 2026

View reviewed changes

keycloak-github-bot Bot mentioned this pull request Sep 8, 2026

Flaky test: org.keycloak.testsuite.broker.KcOidcBrokerTest#loginWithExistingUserWithBruteForceEnabled #52470

Open

ahus1 marked this pull request as ready for review September 8, 2026 09:54

Copilot AI balanced review requested due to automatic review settings September 8, 2026 09:54

ahus1 requested review from a team as code owners September 8, 2026 09:54

Copilot started reviewing on behalf of ahus1 September 8, 2026 09:55 View session