Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Source ID and Fragment ID Overhaul #205

Open
retrodaredevil opened this issue Mar 13, 2024 · 0 comments
Open

Source ID and Fragment ID Overhaul #205

retrodaredevil opened this issue Mar 13, 2024 · 0 comments
Labels
enhancement New feature or request technical

Comments

@retrodaredevil
Copy link
Member

Source ID and Fragment ID have been around a while. The general idea behind them is this:

  • Every SolarThing instance has a unique fragment ID. This helps distinguish a given packet collection from other packet collections that are from different devices
  • Most SolarThing instances are configured to use the "default" Source ID. This can be used to separate unrelated devices.

There are a couple of problems with how these are used across SolarThing.

First, Source IDs are not configurable after the fact. This is not good because it means once data is uploaded with a particular source ID, it cannot be changed. Additionally, source IDs introduce unneeded complexity when configuring SolarThing because of how infrequently a value other than "default" is used.

Fragment IDs work pretty well, but the value of the fragment ID actually determines how it is used in SolarThing. This is not well documented, and probably isn't what we want, since it is not changeable after the fact. Its main use is shown here:

public static List<FragmentedPacketGroup> mergePackets(List<? extends InstancePacketGroup> instancePacketGroups, long maxTimeDistance, Long masterIdIgnoreDistance, Comparator<Integer> fragmentIdComparator){
if (instancePacketGroups.isEmpty()) {
throw new IllegalArgumentException("instancePacketGroups cannot be empty!");
}
Map<Integer, List<InstancePacketGroup>> fragmentMap = new HashMap<>();
for(InstancePacketGroup packetGroup : instancePacketGroups){ // this for loop initializes fragmentMap
final int fragmentId = packetGroup.getFragmentId();
List<InstancePacketGroup> fragmentList = fragmentMap.get(packetGroup.getFragmentId());
if(fragmentList == null){
fragmentList = new ArrayList<>();
fragmentMap.put(fragmentId, fragmentList);
}
fragmentList.add(packetGroup);
}
final List<Integer> fragmentIds;
{ // initialize fragmentIds
SortedSet<Integer> fragmentIdsSet = new TreeSet<>(fragmentIdComparator);
fragmentIdsSet.addAll(fragmentMap.keySet());
fragmentIds = new ArrayList<>(fragmentIdsSet); // now this is sorted
}
// Create an ordered set so that whenever something is added to it, it gets ordered correctly
TreeSet<FragmentedPacketGroup> packetGroups = new TreeSet<>(Comparator.comparingLong(PacketGroup::getDateMillis));
addToPacketGroups(
maxTimeDistance, masterIdIgnoreDistance,
Long.MIN_VALUE, Long.MAX_VALUE,
fragmentIds,
fragmentMap,
packetGroups
);
return new ArrayList<>(packetGroups);
}


I think the solution should be that Source IDs are removed altogether and replaced with some sort of grouping that can be configured in the metadata of the database. We should find a better solution than mergePackets that doesn't involve selecting the lowest fragment ID to be the master.

Not sure when I'll get around to this, but these are my thoughts on it for now.

@retrodaredevil retrodaredevil added the enhancement New feature or request label Mar 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request technical
Projects
None yet
Development

No branches or pull requests

1 participant