SnappySession
Related Docs:
object SnappySession
| package sql
Append dataframe to cache table in Spark.
Append dataframe to cache table in Spark.
default storage level is MEMORY_AND_DISK
@todo -> return type?
Convert a BaseRelation created for external data sources into a DataFrame.
Convert a BaseRelation created for external data sources into a DataFrame.
2.0.0
Interface through which the user may create, drop, alter or query underlying databases, tables, functions etc.
Interface through which the user may create, drop, alter or query underlying databases, tables, functions etc.
2.0.0
Close the session which will be unusable after this call.
Close the session which will be unusable after this call.
Runtime configuration interface for Spark.
Runtime configuration interface for Spark.
This is the interface through which the user can get and set all Spark and Hadoop
configurations that are relevant to Spark SQL. When getting the value of a config,
this defaults to the value set in the underlying SparkContext, if any.
2.0.0
Create approximate structure to query top-K with time series support.
Create approximate structure to query top-K with time series support. Java friendly api.
the qualified name of the top-K structure
the base table of the top-K structure, if any, or null
When set to true it will ignore if a table with the same name is present, else it will throw table exist exception
provide lot more details and examples to explain creating and using TopK with time series
Create approximate structure to query top-K with time series support.
Create approximate structure to query top-K with time series support.
the qualified name of the top-K structure
the base table of the top-K structure, if any
When set to true it will ignore if a table with the same name is present, else it will throw table exist exception
provide lot more details and examples to explain creating and using TopK with time series
Create approximate structure to query top-K with time series support.
Create approximate structure to query top-K with time series support. Java friendly api.
the qualified name of the top-K structure
the base table of the top-K structure, if any, or null
When set to true it will ignore if a table with the same name is present, else it will throw table exist exception
provide lot more details and examples to explain creating and using TopK with time series
Create approximate structure to query top-K with time series support.
Create approximate structure to query top-K with time series support.
the qualified name of the top-K structure
the base table of the top-K structure, if any
When set to true it will ignore if a table with the same name is present, else it will throw table exist exception
provide lot more details and examples to explain creating and using TopK with time series
Applies a schema to a List of Java Beans.
Applies a schema to a List of Java Beans.
WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.
1.6.0
Applies a schema to an RDD of Java Beans.
Applies a schema to an RDD of Java Beans.
WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.
2.0.0
Applies a schema to an RDD of Java Beans.
Applies a schema to an RDD of Java Beans.
WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.
2.0.0
:: DeveloperApi ::
Creates a DataFrame from a java.util.List containing Rows using the given schema.
:: DeveloperApi ::
Creates a DataFrame from a java.util.List containing Rows using the given schema.
It is important to make sure that the structure of every Row of the provided List matches
the provided schema. Otherwise, there will be runtime exception.
2.0.0
:: DeveloperApi ::
Creates a DataFrame from a JavaRDD containing Rows using the given schema.
:: DeveloperApi ::
Creates a DataFrame from a JavaRDD containing Rows using the given schema.
It is important to make sure that the structure of every Row of the provided RDD matches
the provided schema. Otherwise, there will be runtime exception.
2.0.0
:: DeveloperApi ::
Creates a DataFrame from an RDD containing Rows using the given schema.
:: DeveloperApi ::
Creates a DataFrame from an RDD containing Rows using the given schema.
It is important to make sure that the structure of every Row of the provided RDD matches
the provided schema. Otherwise, there will be runtime exception.
Example:
import org.apache.spark.sql._ import org.apache.spark.sql.types._ val sparkSession = new org.apache.spark.sql.SparkSession(sc) val schema = StructType( StructField("name", StringType, false) :: StructField("age", IntegerType, true) :: Nil) val people = sc.textFile("examples/src/main/resources/people.txt").map( _.split(",")).map(p => Row(p(0), p(1).trim.toInt)) val dataFrame = sparkSession.createDataFrame(people, schema) dataFrame.printSchema // root // |-- name: string (nullable = false) // |-- age: integer (nullable = true) dataFrame.createOrReplaceTempView("people") sparkSession.sql("select name from people").collect.foreach(println)
2.0.0
:: Experimental ::
Creates a DataFrame from a local Seq of Product.
:: Experimental ::
Creates a DataFrame from a local Seq of Product.
2.0.0
:: Experimental ::
Creates a DataFrame from an RDD of Product (e.g.
:: Experimental ::
Creates a DataFrame from an RDD of Product (e.g. case classes, tuples).
2.0.0
:: Experimental :: Creates a DataFrame from an RDD of Product (e.g.
:: Experimental :: Creates a DataFrame from an RDD of Product (e.g. case classes, tuples). This method handles generic array datatype like Array[Decimal]
:: Experimental :: Creates a Dataset from an RDD of a given type.
:: Experimental ::
Creates a Dataset from an RDD of a given type. This method requires an
encoder (to convert a JVM object of type T to and from the internal Spark SQL representation)
that is generally created automatically through implicits from a SparkSession, or can be
created explicitly by calling static methods on Encoders.
2.0.0
:: Experimental :: Creates a Dataset from a java.util.List of a given type.
:: Experimental ::
Creates a Dataset from a java.util.List of a given type. This method requires an
encoder (to convert a JVM object of type T to and from the internal Spark SQL representation)
that is generally created automatically through implicits from a SparkSession, or can be
created explicitly by calling static methods on Encoders.
List<String> data = Arrays.asList("hello", "world"); Dataset<String> ds = spark.createDataset(data, Encoders.STRING());
2.0.0
:: Experimental :: Creates a Dataset from a local Seq of data of a given type.
:: Experimental ::
Creates a Dataset from a local Seq of data of a given type. This method requires an
encoder (to convert a JVM object of type T to and from the internal Spark SQL representation)
that is generally created automatically through implicits from a SparkSession, or can be
created explicitly by calling static methods on Encoders.
import spark.implicits._ case class Person(name: String, age: Long) val data = Seq(Person("Michael", 29), Person("Andy", 30), Person("Justin", 19)) val ds = spark.createDataset(data) ds.show() // +-------+---+ // | name|age| // +-------+---+ // |Michael| 29| // | Andy| 30| // | Justin| 19| // +-------+---+
2.0.0
Create an external table from the given path based on a data source, a schema and a set of options.
Create an external table from the given path based on a data source, a schema and a set of options. For inbuilt relation providers like row or column tables, use createTable.
Name of the table
Provider name such as 'CSV', 'PARQUET' etc.
Table schema
Properties for table creation. See options list for different tables. https://github.com/TIBCOSoftware/snappydata /blob/master/docs/programming_guide/tables_in_snappydata.md
When set to true it will ignore if a table with the same name is present, else it will throw table exist exception
DataFrame for the table
Create an external table from the given path based on a data source, a schema and a set of options.
Create an external table from the given path based on a data source, a schema and a set of options. For inbuilt relation providers like row or column tables, use createTable.
Name of the table
Provider name such as 'CSV', 'PARQUET' etc.
Table schema
Properties for table creation. See options list for different tables. https://github.com/TIBCOSoftware/snappydata /blob/master/docs/programming_guide/tables_in_snappydata.md
When set to true it will ignore if a table with the same name is present, else it will throw table exist exception
DataFrame for the table
Create an external table from the given path based on a data source, a schema and a set of options.
Create an external table from the given path based on a data source, a schema and a set of options. For inbuilt relation providers like row or column tables, use createTable.
Name of the table
Provider name such as 'CSV', 'PARQUET' etc.
Properties for table creation
When set to true it will ignore if a table with the same name is present, else it will throw table exist exception
DataFrame for the table
Create an external table from the given path based on a data source, a schema and a set of options.
Create an external table from the given path based on a data source, a schema and a set of options. For inbuilt relation providers like row or column tables, use createTable.
Name of the table
Provider name such as 'CSV', 'PARQUET' etc.
Properties for table creation
When set to true it will ignore if a table with the same name is present, else it will throw table exist exception
DataFrame for the table
Create an index on a table.
Create an index on a table.
Index name which goes in the catalog
Fully qualified name of table on which the index is created.
Columns on which the index has to be created with the direction of sorting. Direction can be specified as None.
Options for indexes. For e.g. column table index - ("COLOCATE_WITH"->"CUSTOMER"). row table index - ("INDEX_TYPE"->"GLOBAL HASH") or ("INDEX_TYPE"->"UNIQUE")
Create an index on a table.
Create an index on a table.
Index name which goes in the catalog
Fully qualified name of table on which the index is created.
Columns on which the index has to be created along with the sorting direction.
Sorting direction for indexColumns. The direction of index will be ascending if value is true and descending when value is false. The values in this list must exactly match indexColumns list. Direction can be specified as null in which case ascending is used.
Options for indexes. For e.g. column table index - ("COLOCATE_WITH"->"CUSTOMER"). row table index - ("INDEX_TYPE"->"GLOBAL HASH") or ("INDEX_TYPE"->"UNIQUE")
Create a stratified sample table.
Create a stratified sample table. Java friendly version.
the qualified name of the table
the base table of the sample table, if any, or null
schema of the table
sampling options like QCS, reservoir size etc.
When set to true it will ignore if a table with the same name is present, else it will throw table exist exception
provide lot more details and examples to explain creating and using sample tables with time series and otherwise
Create a stratified sample table.
Create a stratified sample table.
the qualified name of the table
the base table of the sample table, if any
schema of the table
sampling options like QCS, reservoir size etc.
When set to true it will ignore if a table with the same name is present, else it will throw table exist exception
provide lot more details and examples to explain creating and using sample tables with time series and otherwise
Create a stratified sample table.
Create a stratified sample table. Java friendly version.
the qualified name of the table
the base table of the sample table, if any, or null
sampling options like QCS, reservoir size etc.
When set to true it will ignore if a table with the same name is present, else it will throw table exist exception
provide lot more details and examples to explain creating and using sample tables with time series and otherwise
Create a stratified sample table.
Create a stratified sample table.
the qualified name of the table
the base table of the sample table, if any
sampling options like QCS, reservoir size etc.
When set to true it will ignore if a table with the same name is present, else it will throw table exist exception
provide lot more details and examples to explain creating and using sample tables with time series and otherwise
Creates a SnappyData managed JDBC table which takes a free format ddl string.
Creates a SnappyData managed JDBC table which takes a free format ddl string. The ddl string should adhere to syntax of underlying JDBC store. SnappyData ships with inbuilt JDBC store, which can be accessed by Row format data store. The option parameter can take connection details.
val props = Map( "url" -> s"jdbc:derby:$path", "driver" -> "org.apache.derby.jdbc.EmbeddedDriver", "poolImpl" -> "tomcat", "user" -> "app", "password" -> "app" ) val schemaDDL = "(OrderId INT NOT NULL PRIMARY KEY,ItemId INT, ITEMREF INT)" snappyContext.createTable("jdbcTable", "jdbc", schemaDDL, props)
Any DataFrame of the same schema can be inserted into the JDBC table using DataFrameWriter API.
e.g.
case class Data(col1: Int, col2: Int, col3: Int) val data = Seq(Data(1, 2, 3), Data(7, 8, 9), Data(9, 2, 3), Data(4, 2, 3), Data(5, 6, 7)) val dataDF = snc.createDataset(data)(Encoders.product) dataDF.write.insertInto("jdbcTable")
Name of the table
Provider name such as 'COLUMN', 'ROW' etc.
Table schema as a string interpreted by provider
Properties for table creation. See options list for different tables. https://github.com/TIBCOSoftware/snappydata /blob/master/docs/programming_guide/tables_in_snappydata.md
When set to true it will ignore if a table with the same name is present, else it will throw table exist exception
DataFrame for the table
Creates a SnappyData managed JDBC table which takes a free format ddl string.
Creates a SnappyData managed JDBC table which takes a free format ddl string. The ddl string should adhere to syntax of underlying JDBC store. SnappyData ships with inbuilt JDBC store, which can be accessed by Row format data store. The option parameter can take connection details.
val props = Map( "url" -> s"jdbc:derby:$path", "driver" -> "org.apache.derby.jdbc.EmbeddedDriver", "poolImpl" -> "tomcat", "user" -> "app", "password" -> "app" ) val schemaDDL = "(OrderId INT NOT NULL PRIMARY KEY,ItemId INT, ITEMREF INT)" snappyContext.createTable("jdbcTable", "jdbc", schemaDDL, props)
Any DataFrame of the same schema can be inserted into the JDBC table using DataFrameWriter API.
e.g.
case class Data(col1: Int, col2: Int, col3: Int) val data = Seq(Data(1, 2, 3), Data(7, 8, 9), Data(9, 2, 3), Data(4, 2, 3), Data(5, 6, 7)) val dataDF = snc.createDataset(data)(Encoders.product) dataDF.write.insertInto("jdbcTable")
Name of the table
Provider name such as 'COLUMN', 'ROW' etc.
Table schema as a string interpreted by provider
Properties for table creation. See options list for different tables. https://github.com/TIBCOSoftware/snappydata /blob/master/docs/programming_guide/tables_in_snappydata.md
When set to true it will ignore if a table with the same name is present, else it will throw table exist exception
DataFrame for the table
Creates a SnappyData managed table.
Creates a SnappyData managed table. Any relation providers (e.g. row, column etc) supported by SnappyData can be created here.
case class Data(col1: Int, col2: Int, col3: Int) val props = Map.empty[String, String] val data = Seq(Data(1, 2, 3), Data(7, 8, 9), Data(9, 2, 3), Data(4, 2, 3), Data(5, 6, 7)) val dataDF = snc.createDataset(data)(Encoders.product) snappyContext.createTable(tableName, "column", dataDF.schema, props)
For other external relation providers, use createExternalTable.
Name of the table
Provider name such as 'COLUMN', 'ROW' etc.
Table schema
Properties for table creation. See options list for different tables. https://github.com/TIBCOSoftware/snappydata /blob/master/docs/programming_guide/tables_in_snappydata.md
When set to true it will ignore if a table with the same name is present, else it will throw table exist exception
DataFrame for the table
Creates a SnappyData managed table.
Creates a SnappyData managed table. Any relation providers (e.g. row, column etc) supported by SnappyData can be created here.
case class Data(col1: Int, col2: Int, col3: Int) val props = Map.empty[String, String] val data = Seq(Data(1, 2, 3), Data(7, 8, 9), Data(9, 2, 3), Data(4, 2, 3), Data(5, 6, 7)) val dataDF = snc.createDataset(data)(Encoders.product) snappyContext.createTable(tableName, "column", dataDF.schema, props)
For other external relation providers, use createExternalTable.
Name of the table
Provider name such as 'COLUMN', 'ROW' etc.
Table schema
Properties for table creation. See options list for different tables. https://github.com/TIBCOSoftware/snappydata /blob/master/docs/programming_guide/tables_in_snappydata.md
When set to true it will ignore if a table with the same name is present, else it will throw table exist exception
DataFrame for the table
Creates a SnappyData managed table.
Creates a SnappyData managed table. Any relation providers (e.g. row, column etc) supported by SnappyData can be created here.
val airlineDF = snappyContext.createTable(stagingAirline, "column", Map("buckets" -> "29"))
For other external relation providers, use createExternalTable.
Name of the table
Provider name such as 'COLUMN', 'ROW' etc.
Properties for table creation
When set to true it will ignore if a table with the same name is present, else it will throw table exist exception
DataFrame for the table
Creates a SnappyData managed table.
Creates a SnappyData managed table. Any of the table types (e.g. row, column etc) supported by SnappyData can be created here.
val airlineDF = snappyContext.createTable(stagingAirline, "column", Map("buckets" -> "29"))
For other external relation providers, use createExternalTable.
Name of the table
Provider name such as 'COLUMN', 'ROW' etc.
Properties for table creation
When set to true it will ignore if a table with the same name is present, else it will throw table exist exception
DataFrame for the table
Delete all rows in table that match passed filter expression
Delete all rows in table that match passed filter expression
table name
SQL WHERE criteria to select rows that will be updated
number of rows deleted
Drops an index on a table
Drops an index on a table
Index name which goes in catalog
Drop if exists, else exit gracefully
Drop a SnappyData Policy created by a call to createPolicy.
Drop a SnappyData Policy created by a call to createPolicy.
Policy to be dropped
attempt drop only if the Policy exists
Drop a table created by a call to createTable or createExternalTable.
Drop a table created by a call to createTable or createExternalTable.
table to be dropped
attempt drop only if the table exists
Drop a view.
Drop a view.
name of the view to be dropped
attempt drop only if the view exists
Returns a DataFrame with no rows or columns.
Returns a DataFrame with no rows or columns.
2.0.0
:: Experimental :: Creates a new Dataset of type T containing zero elements.
:: Experimental :: Creates a new Dataset of type T containing zero elements.
2.0.0
:: Experimental :: A collection of methods that are considered experimental, but can be used to hook into the query planner for advanced functionality.
:: Experimental :: A collection of methods that are considered experimental, but can be used to hook into the query planner for advanced functionality.
2.0.0
Get name of a previously registered class using addClass.
Get DictionaryCode for a previously registered variable in ExprCode using addDictionaryCode.
Queries sys.indexes table and returns csv string of primary column names of given table
Queries sys.indexes table and returns csv string of primary column names of given table
schema that the table belongs to
table for which primary keys are to be found
primary keys of given table as csv string
:: Experimental ::
(Scala-specific) Implicit methods available in Scala for converting
common Scala objects into DataFrames.
:: Experimental ::
(Scala-specific) Implicit methods available in Scala for converting
common Scala objects into DataFrames.
val sparkSession = SparkSession.builder.getOrCreate() import sparkSession.implicits._
2.0.0
Insert one or more org.apache.spark.sql.Row into an existing table
Insert one or more org.apache.spark.sql.Row into an existing table
java.util.ArrayList[java.util.ArrayList[_] rows = ... * snSession.insert(tableName, rows)
table name for the insert operation
list of rows to be inserted into the table
number of rows inserted
Insert one or more org.apache.spark.sql.Row into an existing table
Insert one or more org.apache.spark.sql.Row into an existing table
snSession.insert(tableName, dataDF.collect(): _*)
If insert is on a column table then a row insert can trigger an overflow to column store form row buffer. If the overflow fails due to some condition like low memory , then the overflow fails and exception is thrown, but row buffer values are kept as it is. Any user level counter of number of rows inserted might be invalid in such a case.
table name for the insert operation
list of rows to be inserted into the table
number of rows inserted
Creates a DataFrame from an RDD[Row].
Creates a DataFrame from an RDD[Row].
User can specify whether the input rows should be converted to Catalyst rows.
:: Experimental :: An interface to register custom org.apache.spark.sql.util.QueryExecutionListeners that listen for execution metrics.
:: Experimental :: An interface to register custom org.apache.spark.sql.util.QueryExecutionListeners that listen for execution metrics.
2.0.0
Start a new session with isolated SQL configurations, temporary tables, registered functions are isolated, but sharing the underlying SparkContext and cached data.
Start a new session with isolated SQL configurations, temporary tables, registered functions are isolated, but sharing the underlying SparkContext and cached data.
Note: Other than the SparkContext, all shared state is initialized lazily. This method will force the initialization of the shared state to ensure that parent and child sessions are set up with the same shared state. If the underlying catalog implementation is Hive, this will initialize the metastore, which may take some time.
2.0.0
Parses the data type in our internal string representation.
Parses the data type in our internal string representation. The data type string should
have the same format as the one generated by toString in scala.
It is only used by PySpark.
Upsert one or more org.apache.spark.sql.Row into an existing table
Upsert one or more org.apache.spark.sql.Row into an existing table
java.util.ArrayList[java.util.ArrayList[_] rows = ... * snSession.put(tableName, rows)
table name for the put operation
list of rows to be put on the table
number of rows successfully put
Upsert one or more org.apache.spark.sql.Row into an existing table
Upsert one or more org.apache.spark.sql.Row into an existing table
snSession.put(tableName, dataDF.collect(): _*)
table name for the put operation
list of rows to be put on the table
number of rows successfully put
Fetch the topK entries in the Approx TopK synopsis for the specified time interval.
Fetch the topK entries in the Approx TopK synopsis for the specified time interval. See _createTopK_ for how to create this data structure and associate this to a base table (i.e. the full data set). The time interval specified here should not be less than the minimum time interval used when creating the TopK synopsis.
- The topK structure that is to be queried.
start time as string of the format "yyyy-mm-dd hh:mm:ss". If passed as null, oldest interval is considered as the start interval.
end time as string of the format "yyyy-mm-dd hh:mm:ss". If passed as null, newest interval is considered as the last interval.
Optional. Number of elements to be queried. This is to be passed only for stream summary
returns the top K elements with their respective frequencies between two time
provide an example and explain the returned DataFrame. Key is the attribute stored but the value is a struct containing count_estimate, and lower, upper bounds? How many elements are returned if K is not specified?
:: Experimental ::
Creates a Dataset with a single LongType column named id, containing elements
in a range from start to end (exclusive) with a step value, with partition number
specified.
:: Experimental ::
Creates a Dataset with a single LongType column named id, containing elements
in a range from start to end (exclusive) with a step value, with partition number
specified.
2.0.0
:: Experimental ::
Creates a Dataset with a single LongType column named id, containing elements
in a range from start to end (exclusive) with a step value.
:: Experimental ::
Creates a Dataset with a single LongType column named id, containing elements
in a range from start to end (exclusive) with a step value.
2.0.0
:: Experimental ::
Creates a Dataset with a single LongType column named id, containing elements
in a range from start to end (exclusive) with step value 1.
:: Experimental ::
Creates a Dataset with a single LongType column named id, containing elements
in a range from start to end (exclusive) with step value 1.
2.0.0
:: Experimental ::
Creates a Dataset with a single LongType column named id, containing elements
in a range from 0 to end (exclusive) with step value 1.
:: Experimental ::
Creates a Dataset with a single LongType column named id, containing elements
in a range from 0 to end (exclusive) with step value 1.
2.0.0
Returns a DataFrameReader that can be used to read non-streaming data in as a
DataFrame.
Returns a DataFrameReader that can be used to read non-streaming data in as a
DataFrame.
sparkSession.read.parquet("/path/to/file.parquet") sparkSession.read.schema(schema).json("/path/to/file.json")
2.0.0
:: Experimental ::
Returns a DataStreamReader that can be used to read streaming data in as a DataFrame.
:: Experimental ::
Returns a DataStreamReader that can be used to read streaming data in as a DataFrame.
sparkSession.readStream.parquet("/path/to/directory/of/parquet/files") sparkSession.readStream.schema(schema).json("/path/to/directory/of/json/files")
2.0.0
:: DeveloperApi ::
:: DeveloperApi ::
do we need this anymore? If useful functionality, make this private to sql package ... SchemaDStream should use the data source API? Tagging as developer API, for now
State isolated across sessions, including SQL configurations, temporary tables, registered functions, and everything else that accepts a org.apache.spark.sql.internal.SQLConf.
State isolated across sessions, including SQL configurations, temporary tables, registered functions, and everything else that accepts a org.apache.spark.sql.internal.SQLConf.
Set current schema for the session.
Set current schema for the session.
schema name which goes in the catalog
State shared across sessions, including the SparkContext, cached data, listener, and a catalog that interacts with external systems.
State shared across sessions, including the SparkContext, cached data, listener, and a catalog that interacts with external systems.
Executes a SQL query using Spark, returning the result as a DataFrame.
Executes a SQL query using Spark, returning the result as a DataFrame.
The dialect that is used for SQL parsing can be configured with 'spark.sql.dialect'.
2.0.0
A wrapped version of this session in the form of a SQLContext, for backward compatibility.
A wrapped version of this session in the form of a SQLContext, for backward compatibility.
2.0.0
Stop the underlying SparkContext.
Stop the underlying SparkContext.
2.0.0
:: Experimental ::
Returns a StreamingQueryManager that allows managing all the
StreamingQuerys active on this.
:: Experimental ::
Returns a StreamingQueryManager that allows managing all the
StreamingQuerys active on this.
2.0.0
Returns the specified table as a DataFrame.
Returns the specified table as a DataFrame.
2.0.0
Executes some code block and prints to stdout the time taken to execute the block.
Executes some code block and prints to stdout the time taken to execute the block. This is available in Scala only and is used primarily for interactive testing and debugging.
2.1.0
Empties the contents of the table without deleting the catalog entry.
Empties the contents of the table without deleting the catalog entry.
full table name to be truncated
attempt truncate only if the table exists
A collection of methods for registering user-defined functions (UDF).
A collection of methods for registering user-defined functions (UDF).
The following example registers a Scala closure as UDF:
sparkSession.udf.register("myUDF", (arg1: Int, arg2: String) => arg2 + arg1)
The following example registers a UDF in Java:
sparkSession.udf().register("myUDF", new UDF2<Integer, String, String>() { @Override public String call(Integer arg1, String arg2) { return arg2 + arg1; } }, DataTypes.StringType);
Or, to use Java 8 lambda syntax:
sparkSession.udf().register("myUDF", (Integer arg1, String arg2) -> arg2 + arg1, DataTypes.StringType);
2.0.0
The user-defined functions must be deterministic. Due to optimization, duplicate invocations may be eliminated or the function may even be invoked more times than it is present in the query.
Update all rows in table that match passed filter expression
Update all rows in table that match passed filter expression
snappyContext.update("jdbcTable", "ITEMREF = 3" , Row(99) , "ITEMREF" )
table name which needs to be updated
SQL WHERE criteria to select rows that will be updated
A list containing all the updated column values. They MUST match the updateColumn list passed
List of all column names being updated
Update all rows in table that match passed filter expression
Update all rows in table that match passed filter expression
snappyContext.update("jdbcTable", "ITEMREF = 3" , Row(99) , "ITEMREF" )
table name which needs to be updated
SQL WHERE criteria to select rows that will be updated
A single Row containing all updated column values. They MUST match the updateColumn list passed
List of all column names being updated
Adds or updates structured streaming UI tab.
Adds or updates structured streaming UI tab. All session instances have their own SnappyStreamingQueryListener but shares same UI tab.
The version of Spark on which this application is running.
The version of Spark on which this application is running.
2.0.0