Skip to content
This repository was archived by the owner on Mar 24, 2022. It is now read-only.

Getting Started Guide

Jump to bottom
Liquan Pei edited this page Nov 9, 2013 · 19 revisions
# Prerequisite Software The following software is required to build or run the SpringTrader application. ### To Build * Git 1.7.10 or later * Groovy 1.8.1 or later * Java 1.6 or later

To deploy

  • Java 1.6 or later
  • Erlang R14B (required by RabbitMQ)
  • Pivotal tc Server 2.7 or later
  • Pivotal RabbitMQ Server 2.8 or later
  • Pivotal Gemfire XD
  • Pivotal HD 1.1 or Apache Hadoop 2.0

Optional Software

  • Pivotal RabbitMQ Management Plug-in (to view channel/queue stats)
  • VMware vFabric EM4J Console Plug-in (to monitor JVMs running in VMs from the vSphere Web Client)
# Software Installation and Configuration ## Linux (CentOS/RHEL 5/6) ### Install tc-server Please download Pivotal tc-server at http://www.gopivotal.com/pivotal-products/apps/vfabric. ### Install Gemfire XD Link and instruction will be provided. ### Install Erlang RabbitMQ depends on Erlang. Erlang must be installed before RabbitMQ. Details about installing Erlang are available [here](http://www.rabbitmq.com/install-rpm.html) ### Install RabbitMQ Information on Pivotal RabbitMQ installation and configuration are available [here](http://pubs.vmware.com/vfabric51/topic/com.vmware.vfabric.rabbitmq.2.8/index.html). ### Host Configuration Add a definition of "nanodbserver" to your /etc/hosts file. We use this in jdbc url. ``` 127.0.0.1 localhost nanodbserver ```

Define the JAVA_HOME environment variable and add java to your path. This is required by tc Server.

# Use appropriate path on your system
export JAVA_HOME=/usr/java/jdk1.6.0_X
export PATH=$PATH:/usr/java/jdk1.6.0_X/bin

Now that we have all required vFabric Products installed, lets start them.

### Start RabbitMQ Server RabbitMQ rpm added service scripts for root user. We will use that to start it. ``` sudo service rabbitmq-server start ``` ### Start Gemfire XD Assuming you installed Gemfire XD under your home directory run following to create and start a Gemfire XD Locator and Gemfire XD Server. From now, we use $GEMFIREXD to refer the installation directory of Gemfire XD. Place the required jars for Hadoop in $GEMFIREXD/ext-lib, in this way, Gemfire XD will automatically add them to the classpath of sqlf. Essentially, the jars in $HADOOP/share/hadoop, $HADOOP/share/hadoop/common/lib and $HADOOP/share/hdfs/lib need to be copied to $GEMFIREXD/ext-lib. ``` cd $GEMFIREXD mkdir locator1 server1 sqlf locator start -peer-discovery-address=127.0.0.1 -peer-discovery-port=3241 -dir=locator1 -client-port=1527 -client-bind-address=127.0.0.1 sqlf server start -dir=server1 -client-bind-address=127.0.0.1 -client-port=1528 -locators=127.0.0.1[3241] ``` ### Build SpringTraderHD Application #### Clone SpringTraderHD repo

Clone SpringTrader github repo under your home directory. It should create ~/SpringTraderHD folder.

git clone https://github.com/gopivotal/SpringTraderHD
### Setup Gemfire XD Client ``` Copy the sqlfire.jar file from Gemfire XD installation to ~/SpringTraderHD/lib/ cp $GEMFIREXD/lib/sqlfire.jar ~/SpringTraderHD/lib/sqlfire.jar

Copy the sqlfireclient.jar file from Gemfire XD installation to ~/SpringTraderHD/lib/ cp $GEMFIREXD/lib/sqlfireclient.jar ~/SpringTraderHD/lib/sqlfireclient-1.0.3.jar

Copy the sqlfireclient.jar file into tc Server springtrader template cp $GEMFIREXD/lib/sqlfireclient.jar ~/SpringTraderHD/templates/springtrader/lib/sqlfireclient-1.0.3.jar

<a name="lin-gemf"/>
### Setup Gemfire jar
Download gemfire binaries in zip format [here] (https://my.vmware.com/web/vmware/evalcenter?p=gemfire) and unzip it  
Copy the gemfire.jar from lib directory into ~/SpringTraderHD/lib/gemfire.jar

### Build SpringTraderHD

export GRADLE_OPTS='-Xmx1024m -Xms256m -XX:MaxPermSize=512m'

If you are behind proxy, add proxy to GRADLE_OPTS

export GRADLE_OPTS=$GRADLE_OPTS" -Dhttp.proxyHost=http://proxy.vmware.com -Dhttp.proxyPort=3128"


Run gradle build

cd ~/SpringTraderHD ./gradlew clean build release

if you see a test error, you can alternatively skip tests

./gradlew clean build release -x test


This should generate following files under dist folder

ls dist spring-nanotrader-asynch-services-1.0.1.BUILD-SNAPSHOT.war spring-nanotrader-services-1.0.1.BUILD-SNAPSHOT.war spring-nanotrader-web-1.0.1.BUILD-SNAPSHOT.war spring-nanotrader-web-1.0.1.BUILD-SNAPSHOT.tgz DataGenerator.zip


Unzip dist/DataGenerator.zip and load SpringTraderHD schema

cp $GEMFIREXD/lib/sqlfireclient.jar ~/SpringTraderHD/dist/libs/sqlfireclient-1.0.3.jar cd ~/SpringTraderHD/dist unzip DataGenerator.zip ./createGemireXDSchema cd ..


<a name="lin-deploy"/>
### Deploy SpringTraderHD in tc Server 

Add SpringTraderHD template under tc Server. We will apply this template when creating tc Server instance next. The template contains jdbc driver for Gemfire XD as well as resource configuration and default properties to connect tc Server to Gemfire XD. From now, we sue $TCSERVER to refer the installation directory of tc Server.

cp -r ~/SpringTraderHD/templates/springtrader $TCSERVER/templates


Create a tc Server instance (with instance name = springtrader). Following will create an instance named springtrader.

cd $TCSERVER ./tcruntime-instance.sh create springtrader -t springtrader -f templates/springtrader/sqlfire.properties


Copy SpringTrader war files to the instance we just created.

cp ~/SpringTraderHD/dist/spring-nanotrader-asynch-services-1.0.1.BUILD-SNAPSHOT.war $TCSERVER/springtrader/webapps/spring-nanotrader-asynch-services.war cp ~/SpringTraderHD/dist/spring-nanotrader-services-1.0.1.BUILD-SNAPSHOT.war $TCSERVER/springtrader/webapps/spring-nanotrader-services.war cp ~/SpringTraderHD/dist/spring-nanotrader-web-1.0.1.BUILD-SNAPSHOT.war $TCSERVER/springtrader/webapps/spring-nanotrader-web.war


Update JVM_OPTS for tc Server instance

cd $TCSERVER/springtrader/bin

Add following to setenv.sh

JVM_OPTS="-Xmx1024m -Xss192K -XX:MaxPermSize=192m"


Copy the sqlfireclient into the Tomcat/lib directory

Copy the sqlfireclient.jar to the tc Server $TCSERVER/springtrader/lib directory cp ~/SpringTraderHD/lib/sqlfireclient-1.0.3.jar $TCSERVER/lib/sqlfireclient-1.0.3.jar


Start springtrader tc Server instance

./tcruntime-ctl.sh springtrader start


### Generate Sample Data
After the "springtrader" tc Server instance is running, you can optionally load sample data into the database.

cp $GEMFIREXID/lib/sqlfireclient.jar ~/SpringTraderHD/dist/libs/sqlfireclient-1.0.3.jar cd ~/SpringTraderHD/dist ./generateData


With your Web browser, navigate to the following URL to view the SpringTrader login page: [http://localhost:8080/spring-nanotrader-web](http://localhost:8080/spring-nanotrader-web)


<a name="mac"/>
## MAC

<a name="mac-install"/>
### Install Required Softwares on Mac
Use brew to install **groovy** and **RabbitMQ**. Download and install tc Server and SQLFire evaluation versions manually.

brew install groovy brew install rabbitmq

* Install tc Server Developer Edition under your home directory (https://www.vmware.com/tryvmware/p/survey.php?p=tcserver-dev&lp=1)
* Install Gemfire XD under your home directory  
(link will be provided later) 
* Install Pivotal HD or Apache HD 
(Instructions will be provided) 

<a name="mac-sys-conf"/>
### Host Configuration
Add a definition of "nanodbserver" to your /etc/hosts file. We use this in jdbc url.

127.0.0.1 localhost nanodbserver

_Warning: On a MAC (seen on Lion and Snow Leopard), the /etc/hosts file can be reset by VPN connection software. [See here](http://superuser.com/questions/354902/etc-hosts-getting-reset-in-lion)_

Define the JAVA_HOME environment variable and add java to your path. JAVA_HOME is required by tc Server.

export JAVA_HOME=/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home

Note this may not work for OSX Mavericks, please add Java installation library on your machine. It is typically in /Library/Java/JavaVitualMachines/

<a name="lin-start-rmq"/>
### RabbitMQ Startup

rabbitmq-server &


<a name="lin-start-sqlf"/>
### Start Gemfire XD 
Assuming you installed Gemfire XD under your home directory run following to create and start a Gemfire XD Locator and Gemfire XD Server. From now, we use $GEMFIREXD to refer the installation directory of Gemfire XD.
Place the required jars for Hadoop in $GEMFIREXD/ext-lib, in this way, Gemfire XD will automatically add them to the classpath of sqlf. Essentially, the jars in $HADOOP/share/hadoop, $HADOOP/share/hadoop/common/lib and $HADOOP/share/hdfs/lib need to be copied to $GEMFIREXD/ext-lib.

export PATH=$PATH:$GEMFIREXD/bin # Add Gemfire XD to path to run sqlf command cd $GEMFIREXD mkdir locator1 server1 sqlf locator start -peer-discovery-address=127.0.0.1 -peer-discovery-port=3241 -dir=locator1 -client-port=1527 -client-bind-address=127.0.0.1 sqlf server start -dir=server1 -client-bind-address=127.0.0.1 -client-port=1528 -locators=127.0.0.1[3241]


<a name="mac-clone"/>
### Clone SpringTraderHD repo

Clone SpringTrader github repo under your home directory. It should create ~/SpringTraderHD folder.

git clone https://github.com/gopivotal/SpringTraderHD


<a name="mac-sqlfc"/>
### Setup Gemfire XD jar

Copy the sqlfire.jar file from Gemfire XD installation to ~/SpringTraderHD/lib/ cp $GEMFIREXD/lib/sqlfire.jar ~/SpringTraderHD/lib/sqlfire.jar

Copy the sqlfireclient.jar file from Gemfire XD installation to ~/SpringTraderHD/lib/ cp $GEMFIREXD/lib/sqlfireclient.jar ~/SpringTraderHD/lib/sqlfireclient-1.0.3.jar

Copy the sqlfireclient.jar file into tc Server springtrader template cp $GEMFIREXD/lib/sqlfireclient.jar ~/SpringTraderHD/templates/springtrader/lib/sqlfireclient-1.0.3.jar


<a name="mac-gemf"/>
### Setup Gemfire jar
Download gemfire binaries in zip format [here] (https://my.vmware.com/web/vmware/evalcenter?p=gemfire) and unzip it  
Copy the gemfire.jar from lib directory into ~/SpringTraderHD/lib/gemfire.jar

<a name="mac-build"/>
### Build springtrader

export GRADLE_OPTS='-Xmx1024m -Xms256m -XX:MaxPermSize=512m'

If you are behind proxy, add proxy to GRADLE_OPTS

export GRADLE_OPTS=$GRADLE_OPTS" -Dhttp.proxyHost=http://proxy.vmware.com -Dhttp.proxyPort=3128"

Run gradle build

cd ~/SpringTraderHD ./gradlew clean build release

if you see a test error, you can alternatively skip tests

./gradlew clean build release -x test

This should generate following files under dist folder

ls dist spring-nanotrader-asynch-services-1.0.1.BUILD-SNAPSHOT.war spring-nanotrader-services-1.0.1.BUILD-SNAPSHOT.war spring-nanotrader-web-1.0.1.BUILD-SNAPSHOT.war spring-nanotrader-web-1.0.1.BUILD-SNAPSHOT.tgz DataGenerator.zip

Unzip dist/DataGenerator.zip and load Gemfire XD schema for SpringTraderHD.

cd ~/SpringTraderHD/dist unzip DataGenerator.zip

Please edit ~/SpringTraderHD/dist/nanotrader-gemfireXD-schema.ddl to change 
"NameNode 'hdfs://localhost:9000'" to your HDFS address as needed.  After the necessary changes, please do the following:

./createGemfireXDschema cd ..


<a name="mac-deploy"/>
### Deploy SpringTrader in tc Server 

Add SpringTraderHD template under tc Server. We will apply this template when creating tc Server instance next. The template contains jdbc driver for Gemfire XD as well as resource configuration and default properties to connect tc Server to Gemfire XD. From now, we sue $TCSERVER to refer the installation directory of tc Server.

cp -r ~/SpringTraderHD/templates/springtrader $TCSERVER/templates


Create a tc Server instance (with instance name = springtrader). Following will create an instance named springtrader.

cd $TCSERVER ./tcruntime-instance.sh create springtrader -t springtrader -f templates/springtrader/sqlfire.properties


Copy springtrader war files to the instance we just created.

cp ~/SpringTraderHD/dist/spring-nanotrader-asynch-services-1.0.1.BUILD-SNAPSHOT.war $TCSERVER/springtrader/webapps/spring-nanotrader-asynch-services.war cp ~/SpringTraderHD/dist/spring-nanotrader-services-1.0.1.BUILD-SNAPSHOT.war $TCSERVER/springtrader/webapps/spring-nanotrader-services.war cp ~/SpringTraderHD/dist/spring-nanotrader-web-1.0.1.BUILD-SNAPSHOT.war $TCSERVER/springtrader/webapps/spring-nanotrader-web.war


Update JVM_OPTS for tc Server instance

cd $TCSERVER/springtrader/bin

Add following to setenv.sh

JVM_OPTS="-Xmx1024m -Xss192K -XX:MaxPermSize=192m"


Copy the sqlfireclient.jar into the Tomcat/lib directory

Copy the sqlfireclient.jar to the tc Server $TCSERVER/lib directory cp ~/SpringTraderHD/lib/sqlfireclient-1.0.3.jar $TCSERVER/springtrader/lib/sqlfireclient-1.0.3.jar


Start springtrader tc Server instance

cd $TCSERVER ./tcruntime-ctl.sh springtrader start


### Generate Sample Data
After the "springtrader" tc Server instance is running, you can optionally load sample data into the database.

cd ~/SpringTraderHD/dist ./generateData


With your Web browser, navigate to the following URL to view the SpringTrader login page: [http://localhost:8080/spring-nanotrader-web](http://localhost:8080/spring-nanotrader-web)

<a name="mac-mapreduce"/>
### Analyze Data in Gemfire XD using Pivotal HD or Apache Hadoop
Copy ~/SpringTraderHD/spring-nanotrader-mapreduce/build/libs/spring-nanotrader-mapreduce-1.0-SNAPSHOT.jar to the machine with Pivotal HD or Apache Hadoop installed. From now, we use $HADOOP to refer the installation directory of Pivotal HD or Apache Hadoop.

export HADOOP_CLASSPATH=$GEMFIREXD/lib/sqlfire.jar:GEMFIREXD/lib/sqlfireclient.jar export LIBJARS=$GEMFIREXD/lib/sqlfire.jar,$GEMFIREXD/lib/sqlfireclient.jar,./spring-nanotrader-mapreduce-1.0-SNAPSHOT.jar

You can then run the MapReduce job with

$HADOOP/bin yarn jar spring-nanotrader-mapreduce-1.0-SNAPSHOT.jar com.gopivotal.mr.TopBusyStock -libjars ${LIBJARS}

You can see the result from the analytics tap in the application or from sqlf. In sqlf, you can see the result 
by running

select * from NANOTRADER.BUSY_STOCK;

Note: NodeManager in Yarn uses the same port number as the tc-server. For now, please stop the tc-server first before starting NodeManager. You can stop tc-server by:

$TCSERVER/tcruntime-ctl.sh springtrader stop


<a name="springtrader-sts"/> 
### Run SpringTraderHD under STS

Update ~/SpringtraderHD/templates/springtrader-sqlfire-sts/lib directory with sqlfireclient-1.0.3.jar
Copy springtrader-sqlfire-sts template directory into tc server templates directory $TCSERVER/templates

In STS, right click on blank area in Servers pane, select New --> Server.
On the first page, "Define a New Server", edit the "Server name" field to be something unique, like "tc Server springtrader"; click Next.
On the 2nd page, "tc Server Configuration", leave the default "Create new instance" selected; click Next.
On the 3rd page, "Create tc Server Instance": give the instance a unique "Name" like "springtrader-sqlfire"
    select desired Templates including "springtrader-sqlfire-sts"
    click Next
On the "Add and Remove" page, select desired resources from the Available and Add them to the Configured column; click Finish.

Use "springtrader-postgres-sts" template instead of "springtrader-sqlfire-sts" if you want to setup postgres as the database

Clone this wiki locally