python (12.9k questions)

javascript (9.2k questions)

reactjs (4.7k questions)

java (4.2k questions)

java (4.2k questions)

c# (3.5k questions)

c# (3.5k questions)

html (3.3k questions)

Questions - sparkr

How to use an undefined list of variables as column names in a SparkDataFrame with SparkR?

I keep progressing in the world of SparkR, and I am now facing an issue which I am unable to resolve. Working on SparkDataFrame manipulations, I may want to update some columns, or aggregate others. I...
test-img

Discus23

r

databricks

azure-databricks

sparkr

Votes: 0

Answers: 1

Latest Answer

You may refer to this answer on how to use lapply in conjunction with other SparkR functions to get what you desire instead of using for loops. Sharing one useful function for using SparkR::agg on a l...
test-img

Vivek Atal

Apply a function to multiple columns of a SparkDataFrame, at once

In basic R, this is how I apply a function to multiple columns at once, using data.table : d <- data.table(V1 = rep(1:2, 4:3), V2 = c(1, 2, 4, 5, 2, 3, 4), V3 = 1:7, V4 = sample(letters, 7)) Cols &...
test-img

Discus23

r

databricks

lapply

sparkr

Votes: 0

Answers: 1

Latest Answer

You can extract the column names as a list using SparkR::colnames function, and then use base::lapply on that list. Note that the function argument inside lapply has to use the columns as a Spark colu...
test-img

Vivek Atal

How to store SparkR result into an R object?

Still new to the world of Azure Databricks, the use of SparkR remains very obscure to me, even for very simple tasks... It took me a very long time to find how to count distinct values, and I'm not su...
test-img

Discus23

r

azure

databricks

azure-databricks

sparkr

Votes: 0

Answers: 2

Latest Answer

The SparkR::sql function returns a SparkDataFrame. In order to use this in R as an R data.frame, you can simply coerce it: as.data.frame(sql("select * from db.mytable"))
test-img

Robert Long

How to access a Databricks database with sparklyr

New in Azure Databricks environment, I discover the packages SparkR and sparklyr. From my notebooks with SparkR, I manage to connect to a database : library(SparkR) DW <- sql("select * from my...
test-img

Discus23

r

databricks

sparkr

sparklyr

Votes: 0

Answers: 1

Latest Answer

If it helps anyone, here's what I found that seems to work. Setting the default data base Read table in default data base library(sparklyr) library(dplyr) sc <- spark_connect(method="databri...
test-img

Discus23

Posts

Questions

Blogs

Jobs