Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create GeoWave Subproject #1542

Merged
merged 38 commits into from
Oct 5, 2016

Conversation

jamesmcclain
Copy link
Member

@jamesmcclain jamesmcclain commented Jun 11, 2016

These changes add a GeoWave subproject, which includes a GeowaveAttributeStore, GeowaveLayerReader, and GeowaveLayerWriter.

  • Missing metadata exception
  • Reduction or elimination of use of asInstanceOf
  • Fix geotools Tests
  • Find work-arounds for bugs
  • Unit Tests
  • GridCoverage2D Kryo Serializer
  • Address Feedback
  • Hillshade Demo

Tests can be run locally and they pass, but there appears to be insufficient memory to bring up GeoWave on Travis:

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000768680000, 51904512, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 51904512 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /home/travis/build/geotrellis/geotrellis/hs_err_pid4764.log
+exit 1
The command ".travis/build-and-test.sh" exited with 1.
cache.2
store build cache
0.00s
2.98snothing changed, not updating cache
Done. Your build exited with 1.

@jamesmcclain jamesmcclain force-pushed the feature/jwm/GeoWave branch 4 times, most recently from 712b4ac to 70520b1 Compare June 13, 2016 18:09
@jamesmcclain jamesmcclain force-pushed the feature/jwm/GeoWave branch 2 times, most recently from 21249d4 to b8501be Compare June 19, 2016 19:03
if (implicitly[ClassTag[K]].toString != "geotrellis.spark.SpatialKey")
throw new Exception("Unsupported Key Type")

/* XXX numPartitions, and filterIndexOnly are currently ignored. */
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In GeoWave, is it always 1 index == 1 tile? If so, then you're not really ignoring filter index only, it's just by default doing either true or false behavior. filterIndexOnly is for when the index has more than one tile stored in the values, so that when we are updating tile entries in a backend we can update the tile information while not losing the surrounding tile info.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand your usage of the word "index" above, then I believe there is one index (one address) for one tile and one tile for one address. Thanks for the clarification on flterIndexOnly.

@jamesmcclain jamesmcclain force-pushed the feature/jwm/GeoWave branch 3 times, most recently from c3f6159 to 3164a3a Compare July 6, 2016 14:28
@jamesmcclain jamesmcclain changed the title [WiP] Create GeoWave Subproject Create GeoWave Subproject Jul 6, 2016
@jamesmcclain jamesmcclain changed the title Create GeoWave Subproject [WiP] Create GeoWave Subproject Jul 7, 2016
@jamesmcclain jamesmcclain force-pushed the feature/jwm/GeoWave branch from 3164a3a to e665d9a Compare July 8, 2016 14:13
@jamesmcclain jamesmcclain changed the title [WiP] Create GeoWave Subproject Create GeoWave Subproject Jul 8, 2016
@jamesmcclain jamesmcclain force-pushed the feature/jwm/GeoWave branch from e665d9a to d81c543 Compare July 8, 2016 17:05
@jamesmcclain jamesmcclain changed the title Create GeoWave Subproject [WiP] Create GeoWave Subproject Jul 14, 2016
@jamesmcclain jamesmcclain changed the title [WiP] Create GeoWave Subproject Create GeoWave Subproject Aug 28, 2016
@jamesmcclain jamesmcclain force-pushed the feature/jwm/GeoWave branch 4 times, most recently from 95f02fe to 27c0035 Compare August 29, 2016 00:28
.map { kv =>
val List(name, zoomStr) = kv.getKey.getRow.toString.split(SEP).toList
LayerId(name, zoomStr.toInt)
}
.toList
.distinct

scanner.close ; retval
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A more safe way to write this would be

  val scanner = connector.createScanner(attributeTable, new Authorizations())
  try {
   scanner.iterator 
        .map { kv =>
          val List(name, zoomStr) = kv.getKey.getRow.toString.split(SEP).toList
          LayerId(name, zoomStr.toInt)
        }
        .toList
        .distinct
  } finally {
    scanner.close()
  }

The rule generally followed with parans or no parans on a method that takes no arguments is that, if the method has side effects, include the parans so that it would never be mistaken for a val

Copy link
Member

@pomadchin pomadchin Aug 31, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lossyrob I can takesimilar changes into account for hbase backend too

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well taken, it looks like Eugene addressed this one.

James McClain and others added 22 commits October 4, 2016 09:15
Includes formatting changes, additional comments, GeoWave-API-change
changes, and removal of unused imports.
Includes GeowaveAttributeStore-related changes and removal of unused
imports.
This serializer delegates to normal Java serialization.
When a precision is specified, the corners and edges of extents which
are close to the splits associated with that precision are snapped to
match the splits exactly.  This is done to work-around difficulties
which emerge when the values are close but not exactly the same
@lossyrob
Copy link
Member

lossyrob commented Oct 5, 2016

Merging!

@lossyrob lossyrob merged commit 644c32b into locationtech:master Oct 5, 2016
@jamesmcclain
Copy link
Member Author

😄

@jamesmcclain jamesmcclain deleted the feature/jwm/GeoWave branch October 5, 2016 20:13
@jamesmcclain jamesmcclain restored the feature/jwm/GeoWave branch October 7, 2016 19:39
@jamesmcclain jamesmcclain deleted the feature/jwm/GeoWave branch October 14, 2016 14:23
@lossyrob lossyrob added this to the 1.0 milestone Oct 18, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants