Home
Schedoscope is a scheduling framework for agile development, testing, (re)loading, and monitoring of your Hadoop data warehouse.
Schedoscope makes the headache go away you are certainly going to get when frequently having to rollout and retroactively apply changes to computation logic and data structures in your data warehouse with traditional ETL job schedulers such as Oozie.
Scheduling with Schedoscope is based on three principles:
-
Goal orientation: with Schedoscope, you specify the views you want and the scheduler takes care that the corresponding data are loaded.
-
Self-sufficiency: Schedoscope has all information about views available: structure, dependencies, transformation logic. The scheduler thus can start out from an empty metastore and create all tables and partitions as data are loaded. Also, metadata management and lineage tracing is trivially as data structure and dependencies are explicitly specified.
-
Reloading is loading: Schedoscope implements measures to automatically detect changes to view structure and computation logic; as it is self-sufficient, it can then automatically recompute potentially outdated views.
Get a glance of what Schedoscope does for you:
Build it:
[~]$ git clone https://github.com/ottogroup/schedoscope.git
[~]$ cd schedoscope
[~/schedoscope]$ MAVEN_OPTS='-XX:MaxPermSize=512m' mvn clean install
Follow the Open Street Map tutorial to install and run Schedoscope in a standard Hadoop distribution image:
Read the View DSL Primer for more information about the capabilities of the Schedoscope DSL:
Read more about how Schedoscope actually performs its scheduling work:
Check out Metascope! It's an add-on to Schedoscope for collaborative metadata management, data discovery and exploration, and data lineage tracing:
We have released Version 0.8.6 as a Maven artifact to our Bintray repository (see Setting Up A Schedoscope Project for an example pom).
This version includes support for field level data lineage - automatically inferred from Hive transformations, declaratively specifyable for other transformations - in Metascope. Also, Metascope lineage graph rendering has been reworked. Extensive documentation to come.
Schedoscope now fails immediately if a driver specified in schedoscope.conf cannot be found on the classpath.
We have released Version 0.8.5 as a Maven artifact to our Bintray repository (see Setting Up A Schedoscope Project for an example pom).
This version adds support for float view fields to JDBC exports
We have released Version 0.8.4 as a Maven artifact to our Bintray repository (see Setting Up A Schedoscope Project for an example pom).
This version removes a race condition the file system driver initialization that seems to have been introduced with CDH-5.10. Also, we have changed the way how we delete and recreate output folders for Map/Reduce transformations to avoid Hive partitions pointing to temporarily non-existing folders.
We have released Version 0.8.3 as a Maven artifact to our Bintray repository (see Setting Up A Schedoscope Project for an example pom).
This version has been built against Cloudera's CDH 5.10.1 client libraries. The test framework no longer artificially sets the storage formats of views under test to text, making testing of Spark jobs writing Parquet files simpler. The robustness of the Schedoscope HTTP service has been improved in face of invalid view parameters.
We have released Version 0.8.2 as a Maven artifact to our Bintray repository (see Setting Up A Schedoscope Project for an example pom).
This version provides significant performance improvements when initializing the scheduling state for a large number of views.
We have released Version 0.8.1 as a Maven artifact to our Bintray repository (see Setting Up A Schedoscope Project for an example pom).
This fixes a critical bug that could result in applying commands to all views in a table and not just the ones addressed. Do not use Release 0.8.0
We have released Version 0.8.0 as a Maven artifact to our Bintray repository (see Setting Up A Schedoscope Project for an example pom).
Schedoscope 0.8.0 includes, among other things:
- significant rework of Schedoscope's actor system that supports testing and uses significantly fewer actors reducing stress for poor Akka;
- support for a lot more Hive storage formats;
- definition of arbitrary Hive table properties / SerDes;
- stability, performance, and UI improvements to Metascope;
- the names of views being transformed appear as the job name in the Hadoop resource manager.
Please note that Metascope's database schema has changed with this release, so back up your database before deploying.
We have released Version 0.7.1 as a Maven artifact to our Bintray repository (see Setting Up A Schedoscope Project for an example pom).
This release includes a fix removing bad default values for the driver setting location for some transformation types. Moreover, it now includes the config setting schedoscope.hadoop.viewDataHdfsRoot which allows one to set a root folder different from /hdp for view table data without having to register a new dbPathBuilder builder function for each view.
Spark transformations, finally! Build views based on Scala and Python Spark 1.6.0 jobs or run your Hive transformations on Spark. Test them using the Schedoscope test framework like any other transformation type. HiveContext is supported.
We have also upgraded Schedoscope's dependencies to CDH-5.8.3. There is catch, though: we had to backport Schedoscope 0.7.0 to Scala 2.10 for compatibility with Cloudera's Spark 1.6.0 dependencies.
We have released Version 0.7.0 as a Maven artifact to our Bintray repository (see Setting Up A Schedoscope Project for an example pom).
Minor improvements to test framework.
We have released Version 0.6.6 as a Maven artifact to our Bintray repository (see Setting Up A Schedoscope Project for an example pom).
The test framework has received some love. There are two new testing styles that can make your tests look prettier and run faster:
- compute a view once and execute multiple tests on its data;
- create the Hive structures for input views and views under test once and load these with different data within each test case saving Hive environment setup overhead and keeping input data and assertions next to each other within each test.
We have released Version 0.6.5 as a Maven artifact to our Bintray repository (see Setting Up A Schedoscope Project for an example pom).
We have factored out Oozie, Pig, and shell transformations and their drivers into separate modules and removed knowledge about which transformation types exist from schedoscope-core. Thus, one can now extend Schedoscope with new tranformation types without touching the core.
We have fixed a bug in the test framework where sorting results with null values yielded a null pointer exception.
We have released Version 0.6.4 as a Maven artifact to our Bintray repository (see Setting Up A Schedoscope Project for an example pom).
We have added:
- simple parallel (S)FTP exporting of views
- the ability to manually assign versions to transformations with
defineVersionin order to avoid unnecessary recomputations in complex cases where the automatic transformation logic change detection generates too many false positives.
We have released Version 0.6.3 as a Maven artifact to our Bintray repository (see Setting Up A Schedoscope Project for an example pom).
We have fixed a security issue with Metascope that allowed non-admin users to edit taxonomies.
We have released Version 0.6.2 as a Maven artifact to our Bintray repository (see Setting Up A Schedoscope Project for an example pom).
Hadoop dependencies have been updated to CDH-5.7.1. A critical bug that could result in no more views transforming while depending views still waiting has been fixed. Reliability of Metascope has been improved.
We have released Version 0.6.1 as a Maven artifact to our Bintray repository (see Setting Up A Schedoscope Project for an example pom).
Hive transformations are no longer submitted via Hive Server 2 to the cluster but directly via the hive-exec library. The reason for this change are stability and resource leakage issues commonly encountered when operating Hive Server 2. Please note that Hive transformations are now issued with hive.auto.convert.join set to false by default to limit heap consumption in Schedoscope due to involuntary local map join operations. Refer to Hive Transformation for more information on how to reenable map joins for queries that need them.
Also: quite a few bug fixes, better error messages when using the CLI client, improved parallelization of JDBC exports.
We have released Version 0.6.0 as a Maven artifact to our Bintray repository (see Setting Up A Schedoscope Project for an example pom).
We have updated the checksumming algorithm for Hive transformations such that changes to comments, settings, and formatting no longer affect the checksum. This should significantly reduce operations worries. However, the checksums of all your Hive queries compared to Release 0.5.0 will change. Take care that you issue a materialization request with [mode RESET_TRANSFORMATION_CHECKSUMS](Scheduling Command Reference) when switching to this version to avoid unwanted view recomputations! Hence the switch of the minor release number.
The test framework now automatically checks whether there is an ON condition for each JOIN clause in your Hive queries. Also, it checks whether each input view you provide in basedOn is also declared as a dependency.
(more)
