VectorType for StructType in Pyspark Schema

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP


VectorType for StructType in Pyspark Schema



I'm reading a parquet file that has the following schema:


df.printSchema()

root
|-- time: integer (nullable = true)
|-- amountRange: integer (nullable = true)
|-- label: integer (nullable = true)
|-- pcaVector: vector (nullable = true)



Now I want to test Pyspark structured streaming and I want to use the same parquet files. The closest schema that I was able to create was using ArrayType, but it doesn't work:


schema = StructType(
[
StructField('time', IntegerType()),
StructField('amountRange', IntegerType()),
StructField('label', IntegerType()),
StructField('pcaVector', ArrayType(FloatType()))

]
)
df_stream = spark.readStream
.format("parquet")
.schema(schema)
.load("/home/user/test_arch/data/fraud/")

Caused by: java.lang.ClassCastException: Expected instance of group converter but got "org.apache.spark.sql.execution.datasources.parquet.ParquetPrimitiveConverter"
at org.apache.parquet.io.api.Converter.asGroupConverter(Converter.java:37)
at org.apache.spark.sql.execution.datasources.parquet.ParquetRowConverter$RepeatedGroupConverter.<init>(ParquetRowConverter.scala:659)
at org.apache.spark.sql.execution.datasources.parquet.ParquetRowConverter.org$apache$spark$sql$execution$datasources$parquet$ParquetRowConverter$$newConverter(ParquetRowConverter.scala:308)



How can I create a schema with VectorType, that seems to exist only for Scala, for the StructType in Pyspark?









By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.
-

Popular posts from this blog

Poland

Image
"Polska" redirects here. For the dance, see Polska (dance). "Rzeczpospolita Polska" redirects here. For other uses, see Rzeczpospolita Polska (disambiguation). For other uses, see Poland (disambiguation). Coordinates: 52°N 20°E  /  52°N 20°E  / 52; 20 Republic of Poland Rzeczpospolita Polska    (Polish) Flag Coat of arms Anthem:  "Mazurek Dąbrowskiego" (English: "Poland Is Not Yet Lost" ) Show globe Show map of Europe Location of   Poland    (dark green) – in Europe    (green & dark grey) – in the European Union    (green)  –  [ Legend ] Capital and largest city Warsaw 52°13′N 21°02′E  /  52.217°N 21.033°E  / 52.217; 21.033 Official languages Polish [1] Recognised regional language Kashubian Ethnic groups (2011 [2] [3] [4] ) 98% Polish [note 1] 2% others or not stated Religion (2011 [5] ) 87.58% Roman Catholic 7.10% Opting out of answer 1.28% Other ...
Read more

Jenkinsfile syntax highlighting in Java project using Intellij Idea

Image
Jenkinsfile syntax highlighting in Java project using Intellij Idea We already tried the approaches as listed below: After having searched the web for many hours on multiple days, we still haven't found a helpful resource on this matter. Thus, it appears to make sense to ask a new question here. We are developing our Java projects in IntelliJ idea and want to integrate our builds with Jenkins. When we create a Jenkinsfile in Idea, we do not get syntax highlighting or auto completion. Since we are new to Jenkins, those features would be really useful to us. How can we make Idea be more supportive with Jenkinsfiles? If there is no way to get syntax highlighting and auto completion for a Jenkinsfile in idea, what other editors would be helpful? Please note: We've already tried the plugin https://github.com/oliverlockwood/jenkinsfile-idea-plugin. When the plugin is activated, the Jenkinsfile is recognized as such, but instead of syntax highlighting we get an error message, please s...
Read more

Slightly complicated web text to variable parsing

Image
Clash Royale CLAN TAG #URR8PPP Slightly complicated web text to variable parsing I'm taking text from a website and parsing it into variables. However, the string I get when I pull the text is a bit complicated. It looks like this on the web... Invoice #: 1267 Date: 4/16/2018 10:44:00 AM PO #: Reference: Countermen: A/A The issue i'm having is that all of this is one string. The string also changes dynamically as some orders have text inputted where others don't. Such as some orders that has every field filled while other orders have almost no field filled. Invoice #: 1267 <br> Date: 4/16/2018 10:44:00 AM <br> PO #: <br> Reference: <br> Countermen: A/A This is what is displayed when I inspect the web element. I want to parse out the information into individual strings and ints for a test and i'm having difficulties dealing with the whole 'dynamic' part of the string as some strings will be longer while some will be shorter. Heres some ima...
Read more