Скачать с ютуб Spark User-defined functions в хорошем качестве

Spark User-defined functions

https://bigdataelearning.com/course/a... https://bigdataelearning.com/courses https://bigdataelearning.com Spark lets us define custom SQL functions and use it in our spark SQL. Those custom SQL functions created by user is known as user-defined functions. For example, lower() is a SQL function that we normally use in our SQL queries to lowercase certain column values. If a new such function is created by a user, it is known as user-defined function. Let’s create a “square” udf that we can use it to square a column value. To create the UDF, I am creating a scala method called “squared”. It takes an integer as a parameter. What this function does is simply returns the squared value of the input. Then we should register the udf. I am registering the “squared” function as an UDF. Spark.udf.register(“udf-name”, actual-function-name) Now, in the spark SQL let’s write a query. We have a dataframe with column name as “value”. It contains list of integers. Here I am applying the “square” user-defined function on the “value” column. This should return the squared value of this column. Let’s display the results, using show command. This returns the squared value results as expected.