D3js and RapidMiner
Just a quick post. I’m experimenting with adding a D3js type of dashboard to this site. I wrote a simple process to pull the closing prices of the S&P500 in RapidMiner Studio and saved a CSV file to my Dropbox. From there it uploads it to the S3 folders and renders a nice chart, using a sample javascript from Techan.js. The chart uses both the Techan.js and D3js libraries which provides for a pretty interface and cool chart. Try zooming and panning around, it’s slick!
I originally wrote a small Pandas python script to do the index price scrapping but it mangles the data-time format, even when I set the format_format=%Y-%m-%d. The RapidMiner process makes it super simple to modify the data-time format, so it can work with D3, but I need to investigate this further.
Below is the RapidMiner process that pulls the Yahoo data, formats it, and writes to a CSV.
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="7.0.001">
<context>
<input/>
<output/>
<macros>
<macro>
<key>symbol</key>
<value>^GSPC</value>
</macro>
</macros>
</context>
<operator activated="true" class="process" compatibility="7.0.001" expanded="true" name="Process">
<process expanded="true">
<operator activated="true" class="quantx1:yahoo_historical_data_extractor" compatibility="1.0.006" expanded="true" height="82" name="Yahoo Historical Stock Data" width="90" x="45" y="34">
<parameter key="I agree to abide by Yahoo's Terms & Conditions on financial data usage" value="true"/>
<parameter key="Quick Stock Ticker Data" value="true"/>
<parameter key="Stock Ticker" value="%{symbol}"/>
<parameter key="select_fields" value="CLOSE|VOLUME|OPEN|DAY_LOW|DAY_HIGH"/>
<parameter key="date_format" value="yyyy-MM-dd"/>
<parameter key="date_start" value="2015-01-01"/>
<parameter key="date_end" value="2016-02-15"/>
</operator>
<operator activated="true" class="date_to_nominal" compatibility="7.0.001" expanded="true" height="82" name="Date to Nominal" width="90" x="179" y="34">
<parameter key="attribute_name" value="Date"/>
<parameter key="date_format" value="dd-MMM-yy"/>
</operator>
<operator activated="true" class="rename" compatibility="7.0.001" expanded="true" height="82" name="Rename" width="90" x="313" y="34">
<parameter key="old_name" value="%{symbol}_OPEN"/>
<parameter key="new_name" value="Open"/>
<list key="rename_additional_attributes">
<parameter key="%{symbol}_DAY_HIGH" value="High"/>
<parameter key="%{symbol}_DAY_LOW" value="Low"/>
<parameter key="%{symbol}_CLOSE" value="Close"/>
<parameter key="%{symbol}_VOLUME" value="Volume"/>
</list>
</operator>
<operator activated="true" class="sort" compatibility="7.0.001" expanded="true" height="82" name="Sort" width="90" x="447" y="34">
<parameter key="attribute_name" value="Date"/>
<parameter key="sorting_direction" value="decreasing"/>
</operator>
<operator activated="true" class="write_csv" compatibility="7.0.001" expanded="true" height="82" name="Write CSV" width="90" x="648" y="34">
<parameter key="csv_file" value="C:\Users\tott_000\Dropbox\neuralmarkettrends.com\d3\SP500\data.csv"/>
<parameter key="column_separator" value=","/>
</operator>
<connect from_op="Yahoo Historical Stock Data" from_port="example set" to_op="Date to Nominal" to_port="example set input"/>
<connect from_op="Date to Nominal" from_port="example set output" to_op="Rename" to_port="example set input"/>
<connect from_op="Rename" from_port="example set output" to_op="Sort" to_port="example set input"/>
<connect from_op="Sort" from_port="example set output" to_op="Write CSV" to_port="input"/>
<connect from_op="Write CSV" from_port="through" to_port="result 1"/>
<portSpacing port="source_input 1" spacing="0"/>
<portSpacing port="sink_result 1" spacing="0"/>
<portSpacing port="sink_result 2" spacing="0"/>
</process>
</operator>
</process>