runpy module

We can use internal runpy to execute different moduls in our project.

This is used in my pyspark project.

import runpy
import sys

if __name__ == '__main__':
    module_name = sys.argv[1]
    function_name = sys.argv[2]
    sys.argv = sys.argv[2:] # this is important for next python entry point
    runpy.run_module(module_name, run_name=function_name)

Now, the spark job can be invoked by

spark-submit submit.py "<module_name>" "<function_name>"

Also, we can wrapper this shell command into a script.

module_name=$1
function_name=$2
spark-submit submit.py "$module_name" "$function_name" "${@:3}"