Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flaky test: test_chatbot #53

Closed
nickjalbert opened this issue Jan 23, 2021 · 2 comments · Fixed by #65
Closed

Flaky test: test_chatbot #53

nickjalbert opened this issue Jan 23, 2021 · 2 comments · Fixed by #65
Assignees
Labels
bug Something isn't working tests Adds or improves testing

Comments

@nickjalbert
Copy link
Contributor

I've noticed test_chatbot occasionally fails. Here's the log from one such failure:

=================================== FAILURES ===================================
_________________________________ test_chatbot _________________________________

capsys = <_pytest.capture.CaptureFixture object at 0x10ee0f320>

    def test_chatbot(capsys):
        import sys
        import time
    
        sys.path.append("example_agents/chatbot")
        from example_agents.chatbot.main import ChatBot
        from example_agents.chatbot.env import MultiChatEnv
    
        env_generator = MultiChatEnv()
        # say something in the room for the agent to hear
        client_env = env_generator()
        client_env.reset()
        running_agent = run_agent(
            ChatBot, env_generator, hz=100, max_iters=40, as_thread=True
        )
        while not running_agent.is_alive():
            pass
        time.sleep(0.1)
        response_txt, _, _, _ = client_env.step("one")
        time.sleep(0.1)
        response_txt, _, _, _ = client_env.step("two")
>       assert response_txt == "one", "chatbot should repeat strings from memory"
E       AssertionError: chatbot should repeat strings from memory
E       assert '' == 'one'
E         + one

test_all.py:87: AssertionError
----------------------------- Captured stdout call -----------------------------
resetting agent buffer for thread 4623322560 
resetting agent buffer for thread 123145336320000 
=============== 1 failed, 5 passed, 1 skipped in 408.29 seconds ================
Error: Process completed with exit code 1.
@nickjalbert nickjalbert added bug Something isn't working tests Adds or improves testing labels Jan 23, 2021
@andyk
Copy link
Contributor

andyk commented Jan 23, 2021

Hmm, it has as some time.sleep()s that might need to be increased. I think the test checks that the chat env is passing messages around correctly, but it depends on the hz passed to run_agent() vs the time.sleep() being set in a way that makes sense.

I recently tweaked them to make the test run faster but it sounds like I might need to think them through more carefully.

@andyk andyk self-assigned this Jan 23, 2021
@andyk andyk linked a pull request Jan 29, 2021 that will close this issue
@andyk
Copy link
Contributor

andyk commented Jan 29, 2021

Fixed by #65

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working tests Adds or improves testing
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants