D3js and RapidMiner

2016-02-12 00:00:00 RapidMiner D3js Tutorials

Just a quick post. I’m experimenting with adding a D3js type of dashboard to this site. I wrote a simple process to pull the closing prices of the S&P500 in RapidMiner Studio and saved a CSV file to my Dropbox. From there it uploads it to the S3 folders and renders a nice chart, using a sample javascript from Techan.js. The chart uses both the Techan.js and D3js libraries which provides for a pretty interface and cool chart. Try zooming and panning around, it’s slick!

I originally wrote a small Pandas python script to do the index price scrapping but it mangles the data-time format, even when I set the format_format=%Y-%m-%d. The RapidMiner process makes it super simple to modify the data-time format, so it can work with D3, but I need to investigate this further.

Below is the RapidMiner process that pulls the Yahoo data, formats it, and writes to a CSV.

        <?xml version="1.0" encoding="UTF-8" standalone="no"?>
        <process version="7.0.001">
          <context>
            <input/>
            <output/>
            <macros>
              <macro>
                <key>symbol</key>
                <value>^GSPC</value>
              </macro>
            </macros>
          </context>
          <operator activated="true" class="process" compatibility="7.0.001" expanded="true" name="Process">
            <process expanded="true">
              <operator activated="true" class="quantx1:yahoo_historical_data_extractor" compatibility="1.0.006" expanded="true" height="82" name="Yahoo Historical Stock Data" width="90" x="45" y="34">
                <parameter key="I agree to abide by Yahoo's Terms &amp; Conditions on financial data usage" value="true"/>
                <parameter key="Quick Stock Ticker Data" value="true"/>
                <parameter key="Stock Ticker" value="%{symbol}"/>
                <parameter key="select_fields" value="CLOSE|VOLUME|OPEN|DAY_LOW|DAY_HIGH"/>
                <parameter key="date_format" value="yyyy-MM-dd"/>
                <parameter key="date_start" value="2015-01-01"/>
                <parameter key="date_end" value="2016-02-15"/>
              </operator>
              <operator activated="true" class="date_to_nominal" compatibility="7.0.001" expanded="true" height="82" name="Date to Nominal" width="90" x="179" y="34">
                <parameter key="attribute_name" value="Date"/>
                <parameter key="date_format" value="dd-MMM-yy"/>
              </operator>
              <operator activated="true" class="rename" compatibility="7.0.001" expanded="true" height="82" name="Rename" width="90" x="313" y="34">
                <parameter key="old_name" value="%{symbol}_OPEN"/>
                <parameter key="new_name" value="Open"/>
                <list key="rename_additional_attributes">
                  <parameter key="%{symbol}_DAY_HIGH" value="High"/>
                  <parameter key="%{symbol}_DAY_LOW" value="Low"/>
                  <parameter key="%{symbol}_CLOSE" value="Close"/>
                  <parameter key="%{symbol}_VOLUME" value="Volume"/>
                </list>
              </operator>
              <operator activated="true" class="sort" compatibility="7.0.001" expanded="true" height="82" name="Sort" width="90" x="447" y="34">
                <parameter key="attribute_name" value="Date"/>
                <parameter key="sorting_direction" value="decreasing"/>
              </operator>
              <operator activated="true" class="write_csv" compatibility="7.0.001" expanded="true" height="82" name="Write CSV" width="90" x="648" y="34">
                <parameter key="csv_file" value="C:\Users\tott_000\Dropbox\neuralmarkettrends.com\d3\SP500\data.csv"/>
                <parameter key="column_separator" value=","/>
              </operator>
              <connect from_op="Yahoo Historical Stock Data" from_port="example set" to_op="Date to Nominal" to_port="example set input"/>
              <connect from_op="Date to Nominal" from_port="example set output" to_op="Rename" to_port="example set input"/>
              <connect from_op="Rename" from_port="example set output" to_op="Sort" to_port="example set input"/>
              <connect from_op="Sort" from_port="example set output" to_op="Write CSV" to_port="input"/>
              <connect from_op="Write CSV" from_port="through" to_port="result 1"/>
              <portSpacing port="source_input 1" spacing="0"/>
              <portSpacing port="sink_result 1" spacing="0"/>
              <portSpacing port="sink_result 2" spacing="0"/>
            </process>
          </operator>
        </process>