The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
- Bump GeoTools version up to 30.x #3521
- Dependecies update & fix "not found: type Serializable" compiler bug #3535
- Update GDAL up to 3.9.x #3540
- Fix reprojection with downsampling for GeotiffRasterSource and tile RDDs. Reprojection outside the valid projection bounds may now throw a GeoAttrsError. #3541
- ConstantTile with NoData: support idempotent CellType conversions #3553
3.7.1 - 2024-01-08
- Add JDK11 cross compilation #3435
- Fix FileRangeReaderProvider parsing URI in windows #3507
- Regrid: force crop to avoid going out of memory #3518
- Fix rounding errors/numerical instability in GridExtent and LayoutTileSource #3520
- GeoTrellisPath.parse should preserve unrecognized uri parameters #3529
3.7.0 - 2023-02-26
- Add RasterSourceRDD.tiledLayerRDD within the geometry intersection #3474
- Expose AWS_REQUEST_PAYER environment variable #3479
- Add union/outer join merger of tiles #3482
- Migration to CE3 and other major dependencies upgrade #3389
- Revert graceful JTS fall back and lazy Circe encoders #3463
- Update Cassandra up to 4.x #3382
- Accumulo update up to 1.10.x #3476
- Fixed Extent.translate #3480
- liftCompletableFuture function fix #3483
- Pass baseTiff to new RasterSource on GeoTiffResampleRasterSource.reproject/convert #3485
Mask
andInverseMask
operations preserve tile cell type #3494
3.6.3 - 2022-07-12
3.6.2 - 2022-04-05
- Dependencies update #3452
- Lazy init Circe codecs in the vector module #3457
- Lazy init Extent and ProjectedExtent Circe codecs #3458
- Add a graceful JtsConfig fallback in case of a shapeless versions mismatch #3459
3.6.1 - 2022-03-12
- Add new shading rules to make GT work with Spark 3.x #3397
- Add Buffer Tile #3419
- RasterSourceRDD.tiledLayerRDD should have a configurable partition transform function #3450
- Disambiguate withBufferTileFocalMethods implicit preserving bin compatibility #3422
- Specialize Grid for Int and Long #3428
- Move GeoWave and GeoMesa subproject to their own repositories #3439
- Use JTS 1.18, GeoTools 25.0 #3437
- ArrayTile.{cols | rows} calls boxing fix #3441
3.6.0 - 2021-04-30
- Add method SpatialIndex#pointsInExtentAsIterable #3349
- Spark 3 & Hadoop 3 Support #3218
- Scala 2.13 cross compilation #2893
- Deprecate method SpatialIndex#traversePointsInExtent #3349
- GDALRasterSource gives segmentation fault when reading rasters with NBITS=1 #3300
- Make JTSConfig load lazy #3369
- GDALWarpOptions incorrect rpc flag #3370
- S3LayerDeleter cannot handle over 1000 objects to delete #3371
- Drop Scala 2.11 cross compilation #3259
- Fix MosaicRasterSource.tileToLayout behavior #3338
- Replace geowave subproject with GeoTrellis/GeoWave data adapter #3364
3.5.2 - 2021-02-01
- Remove explicit unused Scaffeine dependency from projects #3314
- Remove an excessive close after the abort call in S3RangeReader #3324
- Fix sigmoidal contrast calculation #3328
- Separated monoidal component of
vectortile.StrictLayer
#3330
3.5.1 - 2020-11-23
- Fix
Encoder[GeometryCollection]
including subclasses of GeometryCollection twice in the json (MultiPolygon, Multipoint,MultiLinestring) #3167 - Fix
LayoutTileSource
buffer should only be 1/2 a cellsize to avoid going out of bounds and creatingNODATA
values #3302 - Remove unused allocation from CroppedTile #3297
- Fix GeometryCollection::getAll extension method #3295
- Update gdal-warp-bindings v1.1.1 #3303
- gdal-warp-bindings 1.1.1 is a bugfix release that addresses a crash when initializing the bindings on MacOS. See:
3.5.0 - 2020-08-18
- Add alternative RasterizeRDD methods for keyed geometries or features #3271
- Fix NoData handling in the ColorMethods.color function for the RGB and RGBA Multiband Rasters #3278
- Fix renderJpg() color is bluer than expected #3203
- Fix combineDouble of ArrayTile and ConstantTile with non-commutative operator #3257
- Update GDAL up to 3.1 #3279
- Fix GeoTiff writer does not currently support WebMercator projection with no EPSG code set #3281
- Fix Tile combine should union cellTypes #3284
- Fix CRS.fromWKT function throws #3209
3.4.1 - 2020-07-16
- GeoTiffRasterSource is not threadsafe enough #3265
- Fix ShortArrayTileResult result ArrayTile fulfillment #3268
- GeoTiffRasterSource TiledToLayout reads by SpatialKey can produce non-256x256 tiles 3267
- Fix Overviews Read Incorrectly when Per Dataset Masks Present #3269
3.4.0 - 2020-06-19
- Fix GeoTrellisRasterSources to properly pass time though all the internal functions #3226
- Move GDAL overview strategy logger to debug level #3230
- Fix S3RDDLayerReader partitioning #3231
- GDALRasterSource works inconsistenly with BitCellType and ByteCellType #3232
- rasterizeWithValue accepts only topologically valid polygons #3236
- Rasterizer.rasterize should be consistent with rasterizeWithValue #3238
- GeoTrellisRasterSource should return None on empty reads #3240
- ArrayTile equals method always returns true if first elements are NaN #3242
- Fixed resource issue with JpegDecompressor that was causing a "too many open files in the system" exception on many parallel reads of JPEG compressed GeoTiffs. #3249
- Fix MosaicRasterSource, GDALRasterSource and GeoTiffResampleRasterSource behavior #3252
- HttpRangeReader should live outside of the Spark package #3254
- HttpRangeReader moved to `geotrellis.util #3256
- Consistently construct GridExtents with
math.round
#3248
3.3.0 - 2020-04-07
- GeoTrellisPath assumes
file
scheme when none provided #3191 - toStrings overrides to common classes #3217
- GeoTrellisRasterSources legacy and temporal layers support #3179
- Fix
OverviewStrategy
instances now define their own overview selection logic and more accurately port GDAL-Warp -ovr options #3196 - Fix
PolygonRasterizer
failure on some inputs #3160 - Fix GeoTiff Byte and UByte CellType conversions #3189
- Fix incorrect parsing of authority in GeoTrellisPath #3191
- GeoTrellisPath.zoomLevel is now
Option[Int]
->Int
to better indicate that it is a required parameter #3191 - Fix the length of one degree at the equator in metres #3197
- Fix Fix DelaunayRasterizer #3202
- RasterSources resolutions should be consistent across implementations #3210
- Bump gdal-warp-bindings version up to 1.0.0 #3211
- Fixed GDALRasterSource.resample method behavior to respect the passed resampleMethod #3211
- Fix Jackson dependencies #3212
- Fix Rasterizer for polygons containing concavities whith
PixelIsArea
option #3192 - Fix spatial join (Spark) when using different partitioning in left and right RDDs #3175
- Fix Monad instance for
PolygonalSummaryResult
#3221
- Removed duplicate JSON codecs for
TileLayout
#3181 - Removed
private
restriction for TiffTagread
overload #3181
3.2.0 - 2019-11-19
- Preserve NODATA values for double cell types when resampling with Max or Min resampler #3144
- Update dependency versions for Scala 2.12 cross build #3132
- Fix eager evaluation of band min-max in
GDALDataset
#
3.1.0 - 2019-10-25
- Fix
StreamingByteReader
over-allocation when reading past EOF #3138 Tile
,ArrayTile
,ConstantTile
,DelegatingTile
,MultibandTile
,MultibandArrayTile
,RasterRegion
,RasterSource
,MosaicRasterSource
converted fromtrait
toabstract class
#3136
3.0.0 - 2019-10-18
- RasterSources API #3053
geotrellis.layer
package to group functionality related to tiled layers.SpatialKey
,SpaceTimeKey
,TemporalKey
,Bounds
,Boundable
,EmptyBounds
,KeyBounds
,TemporalProjectedExtent
,EmptyBoundsError
,SpatialComponent
,TemporalComponent
,LayoutDefinition
,LayoutScheme
,LayoutLevel
,LocalLayoutScheme
,FloatingLayoutScheme
,ZoomedLayoutScheme
,MapKeyTransform
, etc.
geotrellis.store
package to group interfaces related to saving and reading tiled layers. Implementations save moved to:geotrellis.store.accumulo
geotrellis.store.cassandra
geotrellis.store.hbase
geotrellis.store.hadoop
geotrellis.store.s3
geotrellis.vectortile.MVTFeature
which properly conforms to the MVT 2.0 spec. Specifically,MVTFeature
addsid: Option[Long]
property.RangeReader
SPI produced an instance of the reader fromURI
#2998geotrellis.raster.mapalgebra.focal.ZFactor
to improve slope calculations #3014- A
sparseStitch
method ongeotrellis.layer.stitch.SpatialTileLayoutCollectionStitchMethods
. Note thatSpatialTileLayoutCollectionStitchMethods
now has the additional constraintgeotrellis.raster.prototype.TilePrototypeMethods
on typeV
. This should be transparent for users of thegeotrellis.raster.Tile
andgeotrellis.raster.MultibandTile
types #3017 geotrellis.util.np
package which containslinspace
andpercentile
methods that match NumPy functionality. An implicit class was also added togeotrellis.raster
which provides thepercentile
method forgeotrellis.raster.Tile
#3067- Conversion from raster to features of cells #3117
io.circe
JSON encoders and decoders forGeoTiffInfo
and related classes #3128
geotrellis.raster.summary.polygonal.[Multi]TilePolygonalSummaryHandler
replaced withgeotrellis.raster.summary.polygonal.PolygonalSummary
. Users should expect to implement concrete subclasses ofgeotrellis.raster.summary.GridVisitor
and pass those to the new polygonalSummary methods. There are a number of default implementations provided for simple operations ingeotrellis.raster.summary.visitors
- Polygonal summaries on raster RDDs of
RDD[(SpatialKey, T <: Grid[Int])] with Metadata[TileLayerMetadata[SpatialKey]]
can now be performed with far less boilerplate using the same visitor pattern as the new raster polygonal summary API. SeeRDDPolygonalSummary.scala
for additional details. geotrellis.raster.Grid.dimensions
is nowDimensions[N]
instead of(N, N)
#3124GeoTiffInfo
moved out ofGeoTiffReader
into containing package #3128- Replaced use of internal
Mergable
trait with cats'Semigroup
- The
slope
focal method now requires aZFactor
instance. - Use the AWS S3 SDK v2 instead of v1 #2911.
- The implicit/package structure of the
geotrellis.raster
package has changed such that it's now possible to import almost all features/extensions withimport geotrellis.raster._
#2891 geotrellis.pro4j.WKT
object conversion methods to and from EPSG codes have changed. All now return Options rather than silently unwrap. Methods that convert from EPSG code to WKT string are now prefixed consistently withfromEpsg
and methods that convert from WKT string to EPSG code are now prefixed consistently withtoEpsg
.geotrellis.util.CRS.fromWKT
now returnsOption[CRS]
instead ofCRS
after silently unwrapping an optional internally- The
geotrellis.vectortile.{Layer, VectorTile}
interfaces now usesMVTFeature
instead ofgeotrellis.vector.Feature
.
- The
- Change from
scala-logging
tolog4s
#3116 - Update dependencies #2904.
- Update dependencies #3053.
- Bump ScalaPB version up with some API enhancements #2898.
- Artifacts in Viewshed have been addressed, the pixels/meter calculation has also been improved #2917.
- Fix map{X|Y}ToGrid function behavior that could give a bit incorrect results #2953.
- Fix COG layer update bug related to
COGLayerMetadata
zoom ranges ordering #2922. - Use original
ZoomRanges
on COG layer update #2956. ArrayTile
equality will now check the cols, rows, and cellType of the two tiles #2991.- Fix incorrect Deflate compressor usage in GeoTiff reading #2997.
- Refactor IO thread pool usage #3007.
S3RangeReader
will now optionally read the HEADER of an object #3025.FileRangeReaderProvider
can now handle more types ofURI
s #3034.- Bump
proj4
version to fix multiple performance issues #3039. - Fix
HttpRangeReader
swallows 404 error #3073 - Add a ToSpatial function for the collections API #3082.
- Fix
TIFFTagsReader
to skip unsupported tags #3088. reprojectExtentAsPolygon
should be more deterministic #3083.AmazonS3URI.getKey
now returns an empty string instead of null if no key was provided #3096.- Improved handling of null prefix for
S3AttributeStore
. Prefer use ofS3AttributeStore.apply
to take advantage of this improved handling #3096. - Fix Tiled TIFF BitCellType Segments conversion #3102
- Updated
computeResolution
inIterativeCostDistance
to match Iterative Viewshed #3106
- Scala wrappers of JTS Geometry in
geotrellis.vector
#2932 geotrellis.etl
package has been removed. The code has been archived at https://github.com/geotrellis/spark-etl #2969.- The
S3Client
wrapper that was used to support AWS SDK v1 has been removed in favor of directly using AWS SDK S3Client #2911. geotrellis.s3-testkit
has been removed; testing now depends on an external min.io server which mocks the S3 API. #2911.geotrellis.slick
has been removed. Slick support will now reside ingeotrellis-contrib
#2902.
2.3.0 - 2019-04-18
- Fix Accumulo and HBase
AttributeStore
performance (#2899).
2.2.0 - 2019-01-11
- Extract Proj4J to
org.locationtech.proj4j
(#2846). - Fix {Backend}LayerCopy exception handling (#2860).
- Fix cache level in Ingest objects (#2854).
- Catch a possible precision loss on reprojection to the same projection (#2849).
- Make histogram.merge methods more specific (#2852).
- Fixed elliptical radius check in IDW algorithm (#2844).
- Replace all JavaConversions imports by JavaConverters (#2843).
- Made implicit conversions to/from Raster and ProjectedRaster deprecated (#2834).
- Fix vectorizer bug (#2839).
- Added
mapTile
toProjectedRaster
(#2830). - Downgrade Avro version (#2837).
- Fix error in
ReprojectRasterExtent
(#2825). - Dependency updates (#2813).
- Bump Spark version up to 2.4 (#2823).
- Use a local[2] mode inside TestEnvironment trait (#2821).
- Make IO code agnostic about effects (#2818).
- Make IO code agnostic about effects (#2818).
2.1.0 - 2018-10-02
geotrellis.spark.etl.TemporalIngest
main method (#2709).- Add
bbox
field to all GeoJSON Features (#2811).
TileRDDReproject
now works on RDD ofTileFeature[T, D]
(#2803).TileRDDReproject
now usesReproject.Options.errorThreshold
value (#2803).geotrellis.spark.testkit.TestEnvironment
now includesSparkSession
(#2808).geotrellis.raster.RasterRegionReproject
defaults to approximate resampling forMultibandBandTile
(#2803).Stitcher
type class instance is now defined forTileFeature[T, D]
(#2803).- Fix
GeoTiffSegment
conversion toUByteCellType
andUShortCellType
(#2794). - Fix
GeoTiff[T]
interpretation ofAutoHigherResolution
OverviewStrategy
(#2802). GeoTiff[T].getClosestOverview
method is now public (#2804).GeoTiffOptions.storageMethod
now defaults toTiled
instead ofStriped
(#2810).- GeoTiff
TiffTags
class can now read inverted extents (#2812).
2.0.0 2019-01-11
- All focal operations now except an optional
partitioner
parameter. BufferTiles.apply
methods and thebufferTiles
methods now except an optionalpartitioner
parameter.CollectionLayerReader
now has an SPI interface.ZoomResample
can now be used onMultibandTileLayerRDD
.Partitioner
can be specified in thereproject
methods ofTileLayerRDD
.- Compression
level
of GeoTiffs can be specified in theDeflateCompression
constructor. resampleMethod
parameter has been added toCOGLayerWriter.Options
.- A new type called
LayerType
has been created to help identify the nature of a layer (either Avro or COG). LayerHeader
now have an additional parameter:layerType
.AttributeStore
now has four new methods:layerType
,isCOGLayer
,readCOGLayerAttributes
, andwriteCOGLayerAttributes
.- Kryo serialization of geometry now uses a binary format to reduce shuffle block size.
- Alter
geotrellis.spark.stitch.StitchRDDMethods
to allowRDD[(K, V)]
to be stitched when not all tiles are of the same dimension. - Introduce
Pyramid
class to provide a convenience wrapper for building raster pyramids. - Both
COGValueReader
andOverzoomingCOGValueReader
now have thereadSubsetBands
method which allows users to read in a select number of bands in any order they choose. COGLayerReader
now has thereadSubsetBands
andquerySubsetBands
methods which allow users to read in layers with the desired bands in the order they choose.KeyBounds
now has therekey
method that will rekey the bounds from a source layout to a target layout.- Kryo serialization of geometry now uses a binary format to reduce shuffle block size.
GeoTiffMultibandTile
now has anothercrop
method that takes aGridBounds
and anArray[Int]
that represents the band indices.MultibandTile
now has a new method,cropBands
that takes an Array of band indices and returns a croppedMultibandTile
with the chosen bands.GeoTiff[MultibandTile]
can be written withBandInterleave
, onlyPixelInterleave
previously supported. (#2767)Input.partitionBytes
and it is set to134217728
by default to correspondHadoopGeoTiffRDD default behaviour
.Output.bufferSize
option to set up a custom buffer size for the buffered reprojection.
- The length of the key (the space-filling curve index or address) used for layer reading and writing has been extended from a fixed length of 8 bytes to an arbitrary length. This change affects not only the
geotrellis.spark
package, but all backends (excludinggeotrellis.geowave
andgeotrellis.geomesa
). - Reprojection has improved performance due to one less shuffle stage and lower memory usage.
TileRDDReproject
loses dependency onTileReprojectMethods
in favor ofRasterRegionReproject
- The Ascii draw methods are now method extensions of
Tile
. - Replace
geotrellis.util.Functor
withcats.Functor
. - Specifying the
maxTileSize
for a COGLayer that's to be written is now done viaCOGLayerWriter.Options
which can be passed directly to thewrite
methods. - Specifying the
compression
for a COGLayer that's to be written is now done viaCOGLayerWriter.Options
which can be passed directly to thewrite
methods. - The attribute name for
COGLayerStorageMetadata
is nowmetadata
instead ofcog_metadata
. Scalaz
streams were replaced byfs2
streams.- Refactored
HBaseInstance
, now accepts a plain HadoopConfiguration
object. - Refactored
CassandraInstance
, now accepts agetCluster
function. - Use
pureconfig
to handle all work with configuration files. - Change
TilerMethods.tileToLayout
functions that acceptTileLayerMetadata
as an argument to returnRDD[(K, V)] with Metadata[M]
instead ofRDD[(K, V)]
. - Expose
attributeStore
parameter to LayerReader interface. - Added exponential backoffs in
S3RDDReader
. - Changed
SinglebandGeoTiff
andMultibandGeoTiff
crop function behavior to work properly with cases when extent to crop by doesn't intersect tiff extent. - All classes and objects in the
geowave
package now use the spelling:GeoWave
in their names. TileLayerMetadata.fromRdd
method has been renamed toTileLayerMetadata.fromRDD
.KeyBounds.fromRdd
method has been renamed toKeyBounds.fromRDD
.- Removed implicit conversion from
Raster[T]
toT
(#2771). - Removed
decompress
option fromGeoTiffReader
functions. geotrellis.spark-etl
package is deprecated since GeoTrellis 2.0.Input.maxTileSize
is256
by default to correspondHadoopGeoTiffRDD
default behavior.geotrellis.slick.Projected
has been moved togeotrellis.vector.Projected
- StreamingHistogram.binCount now returns non-zero counts (#2590)
- HilbertSpatialKeyIndex index offset. Existing spatial layers using Hilbert index will need to be updated (#2586)
- Fixed
CastException
that sometimes occurred when reading cached attributes. - Uncompressed
GeoTiffMultibandTiles
will now convert to the correctCellType
. - Calculating the Slope of a
Tile
whentargetCell
isData
will now produce the correct result. - Introduce new hooks into
AttributeStore
trait to allow for better performance in certain queries against catalogs with many layers. GeoTiffReader
can now read tiffs that are missing theNewSubfileType
tag.- Pyramiding code will once again respect resampling method and will now actually reduce shuffle volume by resampling tiles on map side of pyramid operation.
COGLayer
attributes can be accessed via the various read attribute methods inAttributeStore
(iereadMetadata
,readHeader
, etc)- The regex used to match files for the
HadoopLayerAttributeStore
andFileLayerAttributeStore
has been expanded to include more characters. HadoopAttributeStore.availableAttributes
has been fixed so that it'll now list all attribute files.- Allow for simple features to be generated with a specified or random id with geometry stored in the standard field,
the_geom
- Update version of Amazon SDK API to remove deprecation warnings.
- Fixed a bug in incorrect metadata fetch by
COGLayerReader
that could lead to an incorrect data querying. - Cropping RDDs with
clamp=false
now produces correct result. - Fixed tiff reads in case
RowsPerStrip
tiff tag is not defined. - Change aspect result to azimuth, i.e. start from due north and be clockwise.
- COG overviews generated in the
COGLayer.fromLayerRDD
method will now use the passed inResampleMethod
. - Reading a GeoTiff with
streaming
will now work with files that are larger thanjava.lang.Integer.MAX_VALUE
. GeoTiffMultibandTile.crop
will now work with GeoTiffs that have tiled segments and band interleave.GeoTiffMultibandTile.crop
will now returnArrayMultibandTile
(s) with the correct number of bands.- Improved performance of
COGValueReader.readSubsetBands
when reading from S3.
LayerUpdater
with its functionality covered byLayerWriter
(#2663).
1.2.1 - 2018-01-03
- GeoTiffSegmentLayout.getIntersectingSegments bounds checking
- Fix for area of vectorizer that can throw topology exceptions
- Fix Tile.flipHorizontal for floating point tiles
1.2.0 - 2017-11-06
This release cycle saw a regular contributor Simeon Fitch elevated to official Committer status within GeoTrellis.
The team would like to thank him, along with our newest contributors Aaron Santos, Austin Heyne, Christos Charmatzis, Jessica Austin, and @mteldridge for helping make this release possible.
geotrellis.raster
- Deprecation:
GridBounds.size
in favor ofGridBounds.sizeLong
. - Deprecation:
GridBounds.coords
in favor ofGridBounds.coordsIter
. - New:
GridBounds.offset
andGridBounds.buffer
for creating a modifiedGridBounds
from an existing one. - New:
ColorRamps.greyscale: Int => ColorRamp
, which will generate a ramp when given some number of stops. - New:
ConstantTile.fromBytes
to create any type ofConstantTile
from anArray[Byte]
. - New:
Tile.rotate90: Int => Tile
,Tile.flipVertical: Tile
andTile.flipHorizontal: Tile
.
- Deprecation:
geotrellis.vector
- New:
Geometry.isEmpty: Boolean
. This incurs much less overhead than previous ways of determining emptiness. - New:
Line.head
andLine.last
for efficiently grabbing the first or lastPoint
in theLine
.
- New:
geotrellis.spark
- Deprecation: The
LayerUpdater
trait hierarchy. UseLayerWriter.update
orLayerWriter.overwrite
instead. - Deprecation: Every cache provided by
geotrellis.spark.util.cache
. These will be removed in favor of a pluggable cache in 2.0. - New:
SpatialKey.extent: LayoutDefinition => Extent
- New:
ValueReader.attributeStore: AttributeStore
- New:
TileLayerRDD.toSpatialReduce: ((V, V) => V) => TileLayerRDD[SpatialKey]
for smarter folding of 3D tile layers into 2D tile layers. - The often-used
apply
method overloads inMapKeyTransform
have been given more descriptive aliases. - Change: Querying a layer will now produce a result whose metadata will have an
Extent
andKeyBounds
of the queried region and not of the whole layer.
- Deprecation: The
geotrellis.vectortile
(experimental)- New:
VectorTile.toGeoJson
andVectorTile.toIterable
. - Library simplified by assuming the codec backend will always be protobuf.
- New:
Finally, the full marriage of the vector
, raster
, and spark
packages! You can now transform an RDD[Geometry]
into a writable
GeoTrellis layer of (SpatialKey, Tile)
!
val geoms: RDD[Geometry] = ...
val celltype: CellType = ...
val layout: LayoutDefinition = ...
val value: Double = ... /* Value to fill the intersecting pixels with */
val layer: RDD[(SpatialKey, Tile)] with Metadata[LayoutDefinition] =
geoms.rasterize(value, celltype, layout)
In a similar vein to the above, you can now transform an arbitrarily large collection of Geometries into a proper GeoTrellis layer, where the sections of each Geometry are clipped to fit inside their enclosing Extents.
Here we can see a large Line
being clipped into nine sublines. It's
one method call:
import geotrellis.spark._
val layout: LayoutDefinition = ... /* The definition of your grid */
val geoms: RDD[Geometry] = ... /* Result of some previous work */
/* There are likely many clipped Geometries per SpatialKey... */
val layer: RDD[(SpatialKey, Geometry)] = geoms.clipToGrid(layout)
/* ... so we can group them! */
val grouped: RDD[(SpatialKey, Iterable[Geometry])] = layer.groupByKey
If clipping on the Extent boundaries is not what you want, there are ways to customize this. See the ClipToGrid entry in our Scaladocs.
A Viewshed shows "visibility" from some set vantage point, given an Elevation raster. Prior to GeoTrellis 1.2 this was possible at the individual Tile
level but not the Layer (RDD
) level. Now it is.
First, we need to think about the Viewpoint
type:
import geotrellis.spark.viewshed._
val point: Viewpoint(
x = ..., // some coordinate.
y = ..., // some coordinate.
viewHeight = 4000, // 4 kilometres above the surface.
angle = Math.PI / 2, // direction that the "camera" faces (in radians). 0 == east.
fieldOfView = Math.PI / 2, // angular width of the "view port".
altitude = 0 // the height of points you're interested in seeing.
)
In other words:
- x, y, viewHeight: where are we?
- angle: what direction are we looking?
- fieldOfView: how wide are we looking?
- altitude: how high/low is the "target" of our viewing?
Given a Seq[Viewpoint]
(the algorithm supports multiple simultaneous
view points), we can do:
// Recall this common alias:
// type TileLayerRDD[K] = RDD[(K, Tile)] with Metadata[TileLayerMetadata[K]]
val layer: TileLayerRDD[SpatialKey] = ... /* Result of previous work */
val viewshed: TileLayerRDD[SpatialKey] = layer.viewshed(Seq(point))
We use Euclidean Distance to render a collection of points into a heat-map of proximities of some area. Say, of two roads crossing:
Prior to GeoTrellis 1.2, this was possible at the individual Tile
level but not the Layer (RDD
) level. Now it is.
/* Result of previous work. Potentially millions of points per SpatialKey. */
val points: RDD[(SpatialKey, Array[Coordinate])] = ...
val layout: LayoutDefinition = ... /* The definition of your grid */
val layer: RDD[(SpatialKey, Tile)] = points.euclideanDistance(layout)
The following was possible prior to GeoTrellis 1.2:
val layer: TileLayerRDD[SpatialKey] = ...
val polygon: Polgyon = ...
/* The maximum value within some Polygon overlaid on a Tile layer */
val summary: Double = layer.polygonalMaxDouble(polygon)
The above is also now possible for layers keyed by SpaceTimeKey
to
form a "time series":
val layer: TileLayerRDD[SpaceTimeKey] = ...
val polygon: MultiPolygon = ...
/* The maximum value within some Polygonal area at each time slice */
val summary: Map[ZonedDateTime, Double] = layer.maxSeries(polygon)
A GeoTrellis ValueReader
connects to some layer catalog and lets you
read individual values (usually Tiles):
import geotrellis.spark.io.s3._
val store: AttributeStore = ...
val reader: Reader[SpatialKey, Tile] = S3ValueReader(store).reader(LayerId("my-catalog", 10))
val tile: Tile = reader.read(SpatialKey(10, 10))
However .reader
is limited to zoom levels that actually exist for the
given layer. Now you can use .overzoomingReader
to go as deep as you
like:
import geotrellis.raster.resample._
val reader: Reader[SpatialKey, Tile] =
S3ValueReader(store).overzoomingReader(LayerId("my-catalog", 20), Average)
val tile: Tile = reader.read(SpatialKey(1000, 1000))
Have you ever wanted to "redraw" a grid over an established GeoTrellis layer? Say, this 16-tile Layer into a 4-tile one, both of 1024x1024 total pixels:
Prior to GeoTrellis 1.2, there was no official way to do this. Now you
can use .regrid
:
/* The result of some previous work. Say each Tile is 256x256. */
val layer: TileLayerRDD[SpatialKey] = ...
/* "Recut" the tiles so that each one is now 512x512.
* No pixels are gained or lost, save some NODATA on the bottom
* and right edges that may appear for padding purposes.
*/
val regridded: TileLayerRDD[SpatialKey] = layer.regrid(512)
You can also regrid to non-rectangular sizes:
val regridded: TileLayerRDD[SpatialKey] = layer.regrid(tileCols = 100, tileRows = 300)
It's common to find a subset of Tiles in a layer that are touched by
some given Polygon
:
val poly: Polygon = ???
val rdd: TileLayerRDD[SpatialKey] =
layerReader
.query[SpatialKey, Tile, TileLayerMetadata[SpatialKey]](Layer("name", zoom))
.where(Intersects(poly))
.result
Now you can perform this same operation with Line
, MultiLine
, and
even (Polygon, CRS)
to ensure that your Layer and Geometry always
exist in the same projection.
Sometimes you just want to visualize a Tile
without going through the
song-and-dance of rendering it to a .png
. The existing
Tile.asciiDraw
method kind of does that, except its output is all in
numbers.
The new Tile.renderAscii: Palette => String
method fulfills your heart's desire:
import geotrellis.raster._
import geotrellis.raster.io.geotiff._
import geotrellis.raster.render.ascii._
val tile: Tile = SinglebandGeoTiff("path/to/tiff.tiff").tile
// println(tile.renderAscii()) // the default
println(tile.renderAscii(AsciiArtEncoder.Palette.STIPLED))
▚▖
▚▚▜▚▚
▚▖▚▜▚▖▚▚
▜▚▚▚▜▚█▚▜▚█▚
█▚▜▖▜▖▚▚█▚▚▜▚█▖
▚▚█▚▜▚▚▚▚▚▚▚▜▚▚▚▚▚
▚▚▖▚▚▚▚▚█▜▚▚▜▚▚▖▚▖▚▖▚
▚▚▚▚█▚▚▚▚▚██▚▚▚▜▖▖██▚▚▜▚
▚▚█▚▚▚▚▚▚▚▜▚▚▚▚▚▚▜▚█▚▚▚▚▚▚▚
█▚▚▖▚█▚▜▚▚▚▚▖▚▚▚▚▚▚▚▚▚▚▜▚▚▚▚▚▚▖
█▚▚▚▜▚▖▚▚▚▚▚▚▚▚▚▚▚▚▚▚▚▚▚██▖▜▚█▚▚▚
█▚▚██▚▚▚▚▚▚▚▚▖▚▚▚▚▚▚▚▚█▚▚▚▚▚▚▖▖▖▚▚▚▚
█▜▚▚██▜▚▚▚▜▖▚▚▜▚█▜▚▚▚▜▚▖▚▜▚█▚▚▖▚▚▖▚▚▖▖▚▚
▚▚█▚▚▚█▚██▚▚▚▚▚▚▚▚▜▚▚█▜▚▖█▚▚▚▜▚▚▚▚▚▚▜▚█▚█
█▚▜▚▜▚█▚▜▚▚▜▚█▚▚▚▚▚▚▚▚▚▚▚▖▚▖▚▚▖▚█▚█▚▚▚▖█▚
████▚███▚▚▚▚██▚▚▚█▜▚▚▖▚▚▚▖▖▚▚▚▚▚▚▚▚█▚▜▖█
▖█▜▚█▚██▜▖▜▜█▜▜█▜▚▚▚▚▚█▖▚▚▚▚█▚▚▚▚▚▚▜▚▚█▖▜
▚▖██▚▜▚█▚▚▜▜█▜▜▜██▚▚▚▚█▚▚▚▜▖▚▚█▚▖▚▜▚▚▚▖▚█
█▚▚▚▚▜▚██▖██▜▚▚█▚▚▖▚▚▜▚▖▚▖▚▚▚▚▚▖▚▚▖▖▖▚▖▚
▚▚▚█▚▚▚▚▚█▜▚▚████▚█▚▚▚█▚▖▚▚▚▖▚▚█▚▚▖▚▚▚▖▖▖
▚▚▚█▚▚▚▖▖▚▜█▜██▜██▚▚▖██▜▚▜▚█▚▚▚▚▚▚▚▚▖▖▜██
▚▚▚▚▜█▚▚▚▚▚█████▚▜██▚██▚▚▚▚▜▚▖▚█▚▚▖▚▖▚▚█
▚▚▜▚▚▚▚▜▚▜▚▚▚▚▜▚█▚▜█▚██▚██▚▚▚▚▖▚▚▚▚▖▖▚▚▖█
▚▜▚▜▚▚▚▚▚▚█▚▚▚▚▚██▜▜▜███▖▚▚▜█▚▚▖▚█▚▚█▚▖▚
▚▜▚▚▚▚▚▚▚▚▚▚▜▜▜▚▚▖▚▖▚▚▜▜██▜▚██▚▚▚▚▚▚▖▜█▚
▚▚▖▚▚█▚█▚▚▚█▚▖▚▚▚█▚▚▚▚▚▜██▚█▜▚█▚▜▚▚███▜█▜
▚▚▚▜▚▚▚▚▚▚▚▚▚▚▚▖█▚█▚▚▜█▜█▜█▜▚▖▚▚▚██▜▜█▚▜
▚▚▚▚▜▚▚▚▚▚▚▜▚▚▚▚▚▚▖▚█▜▖▖█▚▖▜▖▚▖█▚▖█▚▚▜▚█
▚▚█▚▚█▚▚▜▚▚▚▚▜▚▚▚▚▚▜▚▖▚█▜█▚▜▜▚█▖▜███▜▚▚
▚▚▚▚▚▚▖▜▚█▚▚▚▖▚▚▚▚▚▚▚▚▚▚▚▜█▖▜▜▜█▚▚▚▖▚█▚█
▜▚▚▚█▚▖▚█▚█▚▚█▚▚▚▚▚▚▚▖▚▚▚▜▚▚▚▜▚▖▚▖▚▚▚▚▜▚
▚▚▚▚▖▚█▖█▜▚▚▚▚▚▚▚▚▖▚▚▖▖█▚▜▚▖▚▚▚▚▖▖▚█▚▚▚
▚▚▚▚▚▚▚▚▚█▚▚▚▖▚▚▚█▚▜▚█▚▚▖▜██▚▖▚▚▚▚▚▚▚▚▚▖
▚▚▚▚▚▚▚▖▚▚██▚▚▚▚▚▚▚▚▜▚▚█▚██▚▚▚▚▖▚▚▖▚▚█▜▖
▚▚▚▚▚▚▚▚▚▚▚▚▚█▚▜▚▚▚▜▚▚▖▚▚▚▚▚▜▚▚▚▚▖▚▚▚▚▚
▚██▖▚▚▚▚▚▚▚▚▜▚▚█▚▚▚▚▜▚▚▚▚█▜▖▚▚█▜▜█▜█▚▖▚▖
▚▚▚▖▚▚█▚▚▜███▚▚▚▜▚▚▚▚▚█▚▖▖█▖▚████▜███▚██
▚█▚▚▚▚██▜▚▜▚▜▜▜█▜▚█▚▜▖▜▚▚▚█▚▜█▚▜▚▚▚▚▚▖▖
█▜█▚▚▜▚▜▚▜▜▜▚▚▚▚██▖▖▖▚██▖█▚▜▜▚▚▚▚▚▚▖
▚█▜▜▜▜▜██▚▜▚▚▚▚▚▚▖▜▚▜▚▚▚▜▚█▚▚▖▖▖
██▚▚▚▚▚▚▚▜▚▜▖▚██▜▜▚▖▚▚█▚▚▚▖▜▜
▜▚▚▖▚▚▚▖▚▜▜██▜▜▚█▚▚▜▚▚▜██▚
▚▚█▚▜▚▚█▖▜▚▚▚▖█▚▚█▚▚█▚
█▜▜▚▚▜▜▚▚▚▜█▚▚▚▜█▜█
▚▚▖▚█▖▚▖▜▚▖▚▖▜▚
███▖██▚▖▚▚▚▚
▜▚▚█▚▚▖▖█
▚▖▜█▜▚
▖█▚
Gorgeous.
By adding some additional configuration, you can now use our HDFS Layer Backend to read and write GeoTrellis layers to Microsoft Azure's blob storage.
It's now possible to customize how our S3 backend communicates with S3.
GeoTrellis uses the Java Topology Suite for its vector processing. By default, JTS uses a "floating" PrecisionModel. When writing code that needs to be numerically robust, this default can lead to Topology Exceptions.
You can now use Typesafe Config to configure this to your application's needs. See here for the specifics.
- Kerberos authentication is available for properly configured Accumulo clusters
- Polygonal Summaries for MultibandTiles
- Filter GeoTiffRDDs by Geometry
- Can create ValueReaders via URIs through LayerProvides classes
- Can read/write GeoTiffs with Sinusoidal projections
- Can Resample via Sum operation
- Negative grid bounds bug
- getSignedByteArray BugFix - fixes certain read problems
- Allow Merge Queue To Handle Larger Inputs
- Generate Windows That Conform To GeoTiff Segments
- Removed inefficient LayerFilter check
- Fixed issue with S3 URI not having a key prefix
- Improve S3 makePath function
- Fix S3GeoTiffRDD behavior with some options.
- Allow Contains(Point) query for temporal rdds
- Haversine formula fix
- Use Scaffeine instead of LRU cache in HadoopValueReader
- Fix GeoTiffInfo serialization issues
- Estimate partitions number based on GeoTiff segments
- Estimate partitions number basing on a desired partition size
- Pyramid operation preserves partitioning
- Don't constrain GridBounds size to IntMax x IntMax
- 4-Connected Line Drawing
- Added requirement for CRS implementations to provide a readable toString representation.
- Allow rasterizer to store Z value at double precision
- Changed scheme path file from /User -> current working dir
- Fix CRS parser and proj4
cea
projection support
1.1.0 - 2017-06-22
- Spark Enabled Cost Distance
- Conforming Delaunay Triangulation
- Added a fractional-pixel rasterizer for polygons and multipolygons
- Added collections API mapalgebra local and masking functions
- Added withDefaultNoData method for CellTypes
- Moved Spark TestEnvironment to spark-testkit subproject for usage outside of GeoTrellis
- Add convenience overloads to GeoTiff companion object
- Added matplotlib's Magma, Inferno, Plasma, and Viridis color ramps
- Added library of land use classification color maps.
- Add MGRS encode/decode support to proj4
- Rasters write support to HDFS / S3
- Added Range-based reading of HTTP resources
- Improved the WKT parser that powers the WKT.getEpsgCode method
- Updated the geotrellis-geowave subproject to GeoWave 0.9.3
- Updated the geotrellis-geomesa subproject to GeoMesa 1.2.7
- Use H3 rather than Next Fit when building S3 partitions from paths
- Added delimiter option to S3InputFormat and S3GeoTiffRDD.
- Signed S3 Streaming for GeoTiff reader (HTTP with GET instead of HEAD request)
- Relaxed constraints to improve layer deletion capabilities
- Allow HadoopGeoTiffRDD and S3GeoTiffRDD to maintain additional key information such as file name
- Added API sugar for simplifying construction of AvroRecordCodec
- Make compression optional for Avro encoding and decoding
- Optimization to avoid unspecialized Function3 usage in Hillshade, Slope and Aspect
- Updated multiple dependencies
- Upgraded ScalaPB version for VectorTile
- Added Avro codecs for ProjectedExtent and TemporalProjectedExtent and ConstantTile types
- Repartition in ETL when re-tiling increases layer resolution
- In GeoTiff reader, compute CellSize from TIFF tags
- Improved apply methods for constructing S3RangeReader
- Reorganized handling of CellType.name
- Documentation improvements, including porting the docts to reStructuredText
- Added top-level "Sinusoidal" CRS, commonly used with MODIS
- Added conditional to key bounds decomposition to detect full bounds query in Acccumulo.
- Support for the ability to specify output CRS via proj4 string.
- Fixed issues that made GeoTiff streaming off of S3 slow and broken
- Give a better error message for CRS write failures
- Fix clipping logic during polygon layer query
- Fixed type for CRS authority in NAD83
- Moved JsonFormats for CellSize and CellType to their proper place
- Fixed polygon rasterization for complex polygon test cases
- Fixed issue with FileLayerDeleter
- Fixed issue with logger serialization
- Fixed bug in renderPng that caused incorrect rendering of non-floating-point rasters - Don't allow illegal TileLayouts
- Prevent error from happening during Pyramiding
- Ensure tile columns are not zero when rounding
- Fixed malformed XML error that was happening after failed S3 ingest
- Fix issue with S3LayerDeleter deleting files outside of layer
- Fix TemporalProjectedExtentCodec to handling proj4 strings when CRS isn't available
- Fixed layoutForZoom to allow 0 zoom level
- Fixed MapKeyTransform to deal with points north and west of extent
- Fixed GeoTiff reading for GeoTiffs with model tie point and PixelIsPoint
- Fixed issue with reading tiny (4 pixel or less) GeoTiffs
- Fix usage of IntCachedColorMap in Indexed PNG encoding
- Ensure keyspace exists in CassandraRDDWriter
- Resolved repartitioning issue with HadoopGeoTiffRDD
- Fixed schema for intConstantTileCodec
- In HadoopAttributeStore, get absolute path for attributePath
- In AccumuloLayerDeleter, close batch deleter
- S3InputFormat - bucket names support period and dashes
- Fix TMS scheme min zoom level
- S3AttributeStore now handles ending slashes in prefix.
- Cell type NoData logic for unsigned byte / short not working properly
- CellSize values should not be truncated to integer when parsing from Json.
- Fixes to GeoTiff writing with original LZW compression.
- In ArrayTile.convert, debug instead of warn against floating point data loss.
- Fixes incorrect metadata update in a per-tile reprojection case
- Fix issue with duplicate tiles being read for File and Cassandra backends
- Move to a different Json Schema validator
- S3InputFormat does not filter according to extensions when partitionCount is used
- In S3GeoTiffReader, partitionBytes has no effect if maxTileSize is set
- Fixes typos with rasterizer extension methods
- Fix writing multiband GeoTiff with compression
- Fixed issue with BigTiff vs non-BigTiff offset value packing
While we are trying to stick strictly to SemVer, there are slight API changes in this release. We felt that while this does break SemVer in the strictest sense, the change were not enough to warrant a 2.0 release. Our hope is in the future to be more cognizant of API changes for future releases.
-
Made EPSG capitalization consistent in method names:
- In
geotrellis.proj4.CRS
, changedgetEPSGCode
togetEpsgCode
- In
geotrellis.proj4.io.wkt.WKT
, changedfromEPSGCode
tofromEpsgCode
andgetEPSGCode
togetEpsgCode
- In
-
Changed some internal but publicly visible classes dealing with GeoTiff reading
- Changed
size
tolength
inArraySegmentBytes
- Replaced
foreach
on SegmentBytes withgetSegments
, which the caller can iterate over themselves - Changed
getDecompressedBytes
todecompressGeoTiffSegment
- Changed
-
Changed some internal but publicly visible implicit classes and read methods around TiffTagReader
- Added as an implicit parameter to multiple locations, most publicly in
TiffTagReader.read(byteReader: ByteReader, tagsStartPosition: Long)(implicit ttos: TiffTagOffsetSize)
. Also changed that method from being generic to always taking aLong
offset.
- Added as an implicit parameter to multiple locations, most publicly in
-
Moved some misplaced implicit
JsonFormats
- Moved
CellTypeFormat
andCellSizeFormat
fromgeotrellis.spark.etl.config.json
in thespark-etl
subproject togeotrellis.raster.io.json.Implicits
in theraster
subproject.
- Moved
-
Changed LazyLogger from the
com.typesafe.scalalogging
version to our own version- This shouldn't break any code, but technically is an API change.
1.0.0 - 2017-05-17
- GeoTools support
- Streaming GeoTiff reading #1559
- Windowed GeoTiff ingests into GeoTrellis layers, allowing users to ingest large GeoTiffs #1763
- GeoWave Raster/Vector support (experimental)
- GeoMesa Vector support (experimental)
- Create GeoMesa subproject #1621
- Moved to a JSON-configuration ETL process
- Vector Tile reading and writing, file-based and as GeoTrellis layers in RDDs. #1622
- File Backends
- Collections API #1606
- Add
TileFeature
Type #1429 - Added Focal calculation target type #1601
- Triangulation
- Euclidean distance tiles #1552
- Spark, Scala and Java version version support
- Color correction features:
CollectNeighbors
feature, allowing users to group arbitrary values by the neighbor keys according to their SpatialComponent #1860- Documentation: We moved to ReadTheDocs, and put a lot of work into making our docs significantly better. See them here.
- Documentation improvements: Quickstart, Examples
- Added example for translating from
SpaceTimeKey
toSpatialKey
#1549 - doc-examples subproject; example for tiling to GeoTiff #1564
- Added example for focal operation on multiband layer. #1577
- Projections, Extents, and Layout Definitions doc #1608
- Added example of turning a list of features into GeoJSON #1609
- Example:
ShardingKeyIndex[K]
#1633 - Example:
VoxelKey
#1639
- Added example for translating from
- Introduce ADR concept
- Fixed some markdown docs #1625
parseGeoJson
lives ingeotrellis.vector.io
#1649- Parallelize reads for S3, File, and Cassandra backends #1607
- Kernel Density in Spark
- k-Nearest Neighbors
- Updated slick
- Added GeoTiff read/write support of
TIFFTAG_PHOTOMETRIC
viaGeoTiffOptions
. #1667 - Added ability to read/write color tables for GeoTIFFs encoded with palette photometric interpretation #1802
- Added
ColorMap
to String conversion #1512 - Add split by cols/rows to SplitMethods #1538
- Improved HDFS support #1556
- Added Vector Join operation for Spark #1610
- Added Histograms Over Fractions of RDDs of Tiles #1692
- Add
interpretAs
andwithNoData
methods to Tile #1702 - Changed GeoTiff reader to handle BigTiff #1753
- Added
BreakMap
for reclassification based on range values. #1760 - Allow custom save actions on ETL #1764
- Multiband histogram methods #1784
DelayedConvert
feature, allowing users to delay conversions on tiles until a map or combine operation, so that tiles are not iterated over unnecessarily #1797- Add convenience overloads to GeoTiff companion object #1840
- Fixed GeoTiff bug in reading NoData value if len = 4 #1490
- Add detail to avro exception message #1505
- Fix: The
toSpatial
method gives metadata of typeTileLayerMetadata[SpaceTimeKey]
- Custom
Functor
Typeclass #1643
- Custom
- Allow
Intersects(polygon: Polygon)
in layer query #1644 - Optimize
ColorMap
#1648 - Make regex for s3 URLs handle s3/s3a/s3n #1652
- Fixed metadata handling on surface calculation for tile layer RDDs #1684
- Fixed reading GeoJson with 3d values #1704
- Fix to Bicubic Interpolation #1708
- Fixed: Band tags with values of length > 31 have additional white space added to them #1756
- Fixed NoData bug in tile merging logic #1793
- Fixed Non-Point Pixel + Partial Cell Rasterizer Bug #1804
- metasim
- lokifacio
- aeffrig
- jpolchlo
- jbouffard
- vsimko
- longcmu
- miafg
- PR #1611 Any
RDD
ofTile
s can utilize Polygonal Summary methods. (@fosskers) - PR #1573 New
foreach
forMultibandTile
which maps over each band at once. (@hjaekel) - PR #1600 New
mapBands
method to map more cleanly over the bands of aMultibandTile
.
- PR #1561 Fix to polygon sequence union, account that it can result in NoResult. (1)
- PR #1585 Removed warnings; add proper subtyping to GetComponent and SetComponent identity implicits; fix jai travis breakage. (1)
- PR #1569 Moved RDDLayoutMergeMethods functionality to object. (1)
- PR #1494 Add ETL option to specify upper zoom limit for raster layer ingestion (@mbertrand)
- PR #1571 Fix scallop upgrade issue in spark-etl (@pomadchin)
- PR #1543 Fix to Hadoop LayerMover (@pomadchin)
Special thanks to new contributor @mbertrand!
- PR #1451 Optimize reading from compressed Bit geotiffs (@shiraeeshi)
- PR #1454 Fix issues with IDW interpolation (@lokifacio)
- PR #1457 Store FastMapHistogram counts as longs (@jpolchlo)
- PR #1460 Fixes to user defined float/double CellType parsing (@echeipesh)
- PR #1461 Pass resampling method argument to merge in CutTiles (1)
- PR #1466 Handle Special Characters in proj4j (@jamesmcclain)
- PR #1468 Fix nodata values in codecs (@shiraeeshi)
- PR #1472 Fix typo in MultibandIngest.scala (@timothymschier)
- PR #1478 Fix month and year calculations (@shiraeeshi)
- PR #1483 Fix Rasterizer Bug (@jamesmcclain)
- PR #1485 Upgrade dependencies as part of our LocationTech CQ process (1)
- PR #1487 Handle entire layers of NODATA (@fosskers)
- PR #1493 Added support for int32raw cell types in CellType.fromString (@jpolchlo)
- PR #1496 Update slick (@adamkozuch, @moradology)
- PR #1498 Add ability to specify number of streaming buckets (@moradology)
- PR #1500 Add logic to ensure use of minval/avoid repetition of breaks (@moradology)
- PR #1501 SparkContext temporal GeoTiff format args (@echeipesh)
- PR #1510 Remove dep on cellType when specifying layoutExtent (@fosskers)
- PR #1529 LayerUpdater fix (@pomadchin)
Special thanks to new contributors @fosskers, @adamkozuch, @jpolchlo, @shiraeeshi, @lokifacio!
The long awaited GeoTrellis 0.10 release is here!
It’s been a while since the 0.9 release of GeoTrellis, and there are many significant changes and improvements in this release. GeoTrellis has become an expansive suite of modular components that aide users in the building of geospatial application in Scala, and as always we’ve focused specifically on high performance and distributed computing. This is the first official release that supports working with Apache Spark, and we are very pleased with the results that have come out of the decision to support Spark as our main distributed processing engine. Those of you who have been tuned in for a while know we started with a custom built processing engine based on Akka actors; this original execution engine still exists in 0.10 but is in a deprecated state in the geotrellis-engine subproject. Along with upgrading GeoTrellis to support Spark and handle arbitrarily-sized raster data sets, we’ve been making improvements and additions to core functionality, including adding vector and projection support.
It’s been long enough that release notes, stating what has changed since 0.9, would be quite unwieldy. Instead I put together a list of features that GeoTrellis 0.10 supports. This is included in the README on the GeoTrellis Github, but I will put them here as well. It is organized by subproject, with more basic and core subprojects higher in the list, and the subprojects that rely on that core functionality later in the list, along with a high level description of each subproject.
- Represent a Coordinate Reference System (CRS) based on Ellipsoid, Datum, and Projection.
- Translate CRSs to and from proj4 string representations.
- Lookup CRS's based on EPSG and other codes.
- Transform
(x, y)
coordinates from one CRS to another.
- Provides a scala idiomatic wrapper around JTS types: Point, Line (LineString in JTS), Polygon, MultiPoint, MultiLine (MultiLineString in JTS), MultiPolygon, GeometryCollection
- Methods for geometric operations supported in JTS, with results that provide a type-safe way to match over possible results of geometries.
- Provides a Feature type that is the composition of a geometry and a generic data type.
- Read and write geometries and features to and from GeoJSON.
- Read and write geometries to and from WKT and WKB.
- Reproject geometries between two CRSs.
- Geometric operations: Convex Hull, Densification, Simplification
- Perform Kriging interpolation on point values.
- Perform affine transformations of geometries
- GeometryBuilder for building test geometries
- GeometryMatcher for scalatest unit tests, which aides in testing equality in geometries with an optional threshold.
- Provides types to represent single- and multi-band rasters, supporting Bit, Byte, UByte, Short, UShort, Int, Float, and Double data, with either a constant NoData value (which improves performance) or a user defined NoData value.
- Treat a tile as a collection of values, by calling "map" and "foreach", along with floating point valued versions of those methods (separated out for performance).
- Combine raster data in generic ways.
- Render rasters via color ramps and color maps to PNG and JPG images.
- Read GeoTiffs with DEFLATE, LZW, and PackBits compression, including horizontal and floating point prediction for LZW and DEFLATE.
- Write GeoTiffs with DEFLATE or no compression.
- Reproject rasters from one CRS to another.
- Resample of raster data.
- Mask and Crop rasters.
- Split rasters into smaller tiles, and stitch tiles into larger rasters.
- Derive histograms from rasters in order to represent the distribution of values and create quantile breaks.
- Local Map Algebra operations: Abs, Acos, Add, And, Asin, Atan, Atan2, Ceil, Cos, Cosh, Defined, Divide, Equal, Floor, Greater, GreaterOrEqual, InverseMask, Less, LessOrEqual, Log, Majority, Mask, Max, MaxN, Mean, Min, MinN, Minority, Multiply, Negate, Not, Or, Pow, Round, Sin, Sinh, Sqrt, Subtract, Tan, Tanh, Undefined, Unequal, Variance, Variety, Xor, If
- Focal Map Algebra operations: Hillshade, Aspect, Slope, Convolve, Conway's Game of Life, Max, Mean, Median, Mode, Min, MoransI, StandardDeviation, Sum
- Zonal Map Algebra operations: ZonalHistogram, ZonalPercentage
- Operations that summarize raster data intersecting polygons: Min, Mean, Max, Sum.
- Cost distance operation based on a set of starting points and a friction raster.
- Hydrology operations: Accumulation, Fill, and FlowDirection.
- Rasterization of geometries and the ability to iterate over cell values covered by geometries.
- Vectorization of raster data.
- Kriging Interpolation of point data into rasters.
- Viewshed operation.
- RegionGroup operation.
- Build test raster data.
- Assert raster data matches Array data or other rasters in scalatest.
- Generic way to represent key value RDDs as layers, where the key represents a coordinate in space based on some uniform grid layout, optionally with a temporal component.
- Represent spatial or spatiotemporal raster data as an RDD of raster tiles.
- Generic architecture for saving/loading layers RDD data and metadata
to/from various backends, using Spark's IO API with Space Filling
Curve indexing to optimize storage retrieval (support for Hilbert
curve and Z order curve SFCs). HDFS and local file system are
supported backends by default, S3 and Accumulo are supported
backends by the
geotrellis-s3
andgeotrellis-accumulo
projects, respectively. - Query architecture that allows for simple querying of layer data by spatial or spatiotemporal bounds.
- Perform map algebra operations on layers of raster data, including all supported Map Algebra operations mentioned in the geotrellis-raster feature list.
- Perform seamless reprojection on raster layers, using neighboring tile information in the reprojection to avoid unwanted NoData cells.
- Pyramid up layers through zoom levels using various resampling methods.
- Types to reason about tiled raster layouts in various CRS's and schemes.
- Perform operations on raster RDD layers: crop, filter, join, mask, merge, partition, pyramid, render, resample, split, stitch, and tile.
- Polygonal summary over raster layers: Min, Mean, Max, Sum.
- Save spatially keyed RDDs of byte arrays to z/x/y files into HDFS or the local file system. Useful for saving PNGs off for use as map layers in web maps or for accessing GeoTiffs through z/x/y tile coordinates.
- Utilities around creating spark contexts for applications using GeoTrellis, including a Kryo registrator that registers most types.
- Utility code to create test RDDs of raster data.
- Matching methods to test equality of RDDs of raster data in scalatest unit tests.
- Save and load layers to and from Accumulo. Query large layers efficiently using the layer query API.
Save and load layers to and from Casandra. Query large layers efficiently using the layer query API.
- Save/load raster layers to/from the local filesystem or HDFS using Spark's IO API.
- Save spatially keyed RDDs of byte arrays to z/x/y files in S3. Useful for saving PNGs off for use as map layers in web maps.
- Parse command line options for input and output of ETL (Extract, Transform, and Load) applications
- Utility methods that make ETL applications easier for the user to build.
- Work with input rasters from the local file system, HDFS, or S3
- Reproject input rasters using a per-tile reproject or a seamless reprojection that takes into account neighboring tiles.
- Transform input rasters into layers based on a ZXY layout scheme
- Save layers into Accumulo, S3, HDFS or the local file system.
- Read geometry and feature data from shapefiles into GeoTrellis types using GeoTools.
- Save and load geometry and feature data to and from PostGIS using the slick scala database library.
- Perform PostGIS
ST_
operations in PostGIS through scala.