You already have a working Cassandra server on your development machine: you need a spark+deep bundle, we suggest to create one by running:
#Download spark 2.0.11 download#
Please refer to Stratio's website to download the installer and its documentation.
#Download spark 2.0.11 install#
Install a new cluster using the Stratio installer.Enter /opt/sds and run bin/stratio-deep-shell. Once your VM is up and running you can test Deep using the shell. This distribution will include Stratio's customized Cassandra distribution containing our powerful open-source lucene-based secondary indexes, see Stratio documentation for further information. We also distribute the VM with several preloaded datasets in Cassandra. This VM will work on both Virtualbox and VMWare, and comes with a fully configured distribution that also includes Stratio Deep. Download a pre-configured Stratio platform VM Stratio's BigData platform (SDS).Put Deep to work on a working cassandra + spark cluster. Enter deep-parent subproject and perform: mvn clean install (add -DskipTests to skip tests) Install the project in you local maven repository.In IntelliJ: open the project by selecting the deep-parent POM file To configure a development environment in Eclipse: import as Maven project.MongoDB, we tested the integration with MongoDB versions 2.2, 2.4 y 2.6 using Standalone, Replica Set and Sharded Cluster (for Spark MongoDB integration).Ĭonfigure the development and test environment.Cassandra, we tested versions from 1.2.8 up to 2.0.11 (for Spark Cassandra integration).Support for JDBC has been added in version 0.7.0. Support for Aerospike has been added in version 0.6.0. Support for ElasticSearch has been added in version 0.5.0. We are working on further improvements! ElasticSearch integration You can check out our first steps guide here: We added a few working examples for MongoDB in deep-examples subproject, take a look at: Generic cell API, you do not need to specify the collection's schema or add anything to your POJOs, each document will be transform to an object "Cells".
ORM API, you just have to annotate your POJOs with Deep annotations and magic will begin, you will be able to connect MongoDB with Spark using your own model entities. Support for MongoDB has been added in version 0.3.0.
Spark-MongoDB connector is based on Hadoop-mongoDB.
#Download spark 2.0.11 how to#
Please, refer to the deep-example project README for further information on how to setup a working environment. We encourage you to read the more comprehensive documentation hosted on the Openstratio website.ĭeep comes with an example sub project called 'deep-examples' containing a set of working examples, both in Java and Scala. Please, refer to the Deep API documentation to know more about the Cells and Cell objects. To get the value of column 'address' you can issue a c.getCellByName("address").getCellValue(). Once you get an instance 'c' of the Cells object, This interface is a little bit more cumbersome to work with (see the example below),īut has the advantage that it doesn't require the definition of additional entity classes.Įxample: you have a table called 'users' and you decide to use the 'Cells' interface. The second one is a more generic 'cell' API, that will let developerss work on RDD where a 'Cells' object is a collection of .Cell objects.Ĭolumn metadata is automatically fetched from the data store. Your domain entities must be correctly annotated using Deep annotations (take a look at deep-core example entities in package .entity). This abstraction is quite handy, it will let you work on RDD (under the hood Deep will transparently map Cassandra's columns to entity properties). We call this API the 'entity objects' API. The first one will let developers map Cassandra tables to plain old java objects (POJOs), just like if you were using any other ORM. The integration is not based on the Cassandra's Hadoop interface.ĭeep comes with an user friendly API that lets developers create Spark RDDs mapped to Cassandra column families. We actually support Apache Cassandra, MongoDB, Elastic Search, Aerospike, HDFS, S3 and any database accessible through JDBC, but in the near future we will add support for sever other datastores. tar.gz View on GitHubĭeep is a thin integration layer between Apache Spark and several NoSQL datastores. Deep-spark Connecting Apache Spark with different data stores Download.
0 kommentar(er)
