Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Need to enable correct post-1.4.11 PersistenceIds Akka.Persistence.Query support #130

Open
Aaronontheweb opened this issue Dec 15, 2020 · 3 comments
Labels
bug Something isn't working

Comments

@Aaronontheweb
Copy link
Member

set AllocatesAllPersistenceIDsPublisher => false; so TCK specs pass

we're going to need to, in the future, change this plugin so these specs can pass.

Originally posted by @Aaronontheweb in #129 (comment)

@Aaronontheweb
Copy link
Member Author

The issue is that currently, the PersistenceIds queries in the Akka.Persistence.Azure Journal are implemented using what was originally included inside the SQLite journal, which is bad an doesn't work across multiple nodes:

private readonly HashSet<string> _allPersistenceIds = new HashSet<string>();
private readonly HashSet<IActorRef> _allPersistenceIdSubscribers = new HashSet<IActorRef>();

Right now this works by having the journal track all persitentIds in-memory based on what ids are observed by the journal. Historically, we know that this is extremely memory-inefficient and fragile as it doesn't work for persistenceIds that aren't seen by the current node. This needs to be rewritten to query all ids, authoritatively, from the database instead.

@Arkatufus
Copy link
Contributor

Arkatufus commented Mar 15, 2023

This is not possible to do on Azure because we did not store any ordering "column" in Azure, which is needed to implement the "AllPersistenceId" and "AllTaggedEvents" query

@talismax
Copy link

If I use this persistence plugin in a cluster sharding scenario with Persistence.Query, i.e.

`public void StartListeningForPersistenceIds()
{
    var materializer = Context.System.Materializer();
    var journal = PersistenceQuery
        .Get(Context.System)
        .ReadJournalFor<AzureTableStorageReadJournal>(AzureTableStorageReadJournal.Identifier);

    journal.PersistenceIds().RunForeach(ProcessPersistenceId, materializer);
}`

... will I encounter this bug? If so, any amazing workarounds?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
3 participants