Intro

Mage is an open-source data management platform that helps you clean data and prepare it for training AI/ML models.

What does this do?

The current version of Mage includes a data cleaning UI tool that can run locally on your laptop or can be hosted in your own cloud environment.

Why should I use it?

Using a data cleaning tool enables you to quickly visualize data quality issues, easily fix them, and create repeatable data cleaning pipelines that can be used in production environments (e.g. online re-training, inference, etc).

Table of contents

1. Quick start
2. Features
3. Roadmap
4. Contributing
5. Community

Quick start

Try a demo of Mage in Google Colab.

Install library

Install the most recent released version:

$ pip install mage-ai

Or install the current build on GitHub:

$ pip install git+https://github.com/mage-ai/mage-ai.git

Launch tool

Load your data, connect it to Mage, and launch the tool locally.

From anywhere you can execute Python code (e.g. terminal, Jupyter notebook, etc.), run the following:

import mage_ai
import pandas as pd


df = pd.read_csv('/path_to_data')
mage_ai.connect_data(df, name='name_of_dataset')
mage_ai.launch()

Open http://localhost:5000 in your browser to access the tool locally.

To stop the tool, run this command: mage_ai.kill()

Cleaning data

After building a data cleaning pipeline from the UI, you can clean your data anywhere you can execute Python code:

import mage_ai
import pandas as pd


df = pd.read_csv('/path_to_data')

# Option 1: Clean with pipeline uuid
df_cleaned = mage_ai.clean(df, pipeline_uuid='uuid_of_cleaning_pipeline')

# Option 2: Clean with pipeline config directory path
df_cleaned = mage_ai.clean(df, pipeline_config_path='/path_to_pipeline_config_dir')

Demo video (2 min)

More resources

Here is a 🗺️ step-by-step guide on how to use the tool.

1. Jupyter notebook example
2. Google Colaboratory (Colab) example

Check out the 📚 tutorials to quickly become a master of magic.

Features

1. Data visualizations
2. Reports
3. Cleaning actions
4. Data cleaning suggestions

Data visualizations

Inspect your data using different charts (e.g. time series, bar chart, box plot, etc.).

Here’s a list of available charts.

Reports

Quickly diagnose data quality issues with summary reports.

Here’s a list of available reports.

Cleaning actions

Easily add common cleaning functions to your pipeline with a few clicks. Cleaning actions include imputing missing values, reformatting strings, removing duplicates, and many more.

If a cleaning action you need doesn’t exist in the library, you can write and save custom cleaning functions in the UI.

Here’s a list of available cleaning actions.

Data cleaning suggestions

The tool will automatically suggest different ways to clean your data and improve quality metrics.

Here’s a list of available suggestions.

Roadmap

Big features being worked on or in the design phase.

1. Encoding actions (e.g. one-hot encoding, label hasher, ordinal encoding, embeddings, etc.)
2. Data quality monitoring and alerting
3. Apply cleaning actions to columns and values that match a condition

Here’s a detailed list of 🪲 features and bugs that are in progress or upcoming.

Contributing

We welcome all contributions to Mage; from small UI enhancements to brand new cleaning actions. We love seeing community members level up and give people power-ups!

Check out the 🎁 contributing guide to get started by setting up your development environment and exploring the code base.

Got questions? Live chat with us in Slack

Anything you contribute, the Mage team and community will maintain. We’re in it together!

Community

We love the community of Magers (/ˈmājər/); a group of mages who help each other realize their full potential!

To live chat with the Mage team and community, please join the free Mage Slack channel.

For real-time news and fun memes, check out the Mage Twitter.

To report bugs or add your awesome code for others to enjoy, visit GitHub.

License

See the LICENSE file for licensing information.