Run tests for multiple PySpark versions in CI

[MrPowers]

See here for an example: mrpowers-io/quinn#98

This is a good way we make sure we keep supporting legacy versions of Spark well for a long time.

We always want to support lots of Spark versions.

[Oguzozcn]

Hi, so all I have to do is set up a testing environment, install the necessary versions of PySpark and configure CI on my system to run tests against multiple versions, and then run the existing test suite to ensure it passes, right?

[MrPowers]

@Oguzozcn - Yea, the scope is to run the CI test suite against multiple versions of Spark. This will ensure that the project works for multiple supported Spark versions. Let me know if you'd like me to assign this issue to you.

[Oguzozcn]

Sure, I will try my best.

[MrPowers]

Thanks @Oguzozcn, assigned the issue to you. Feel free to ping me if you need any help.

[MrPowers]

@Oguzozcn - unassigning you cause it looks like this didn't ever get worked on.