ETL & Data Sourcing

​

​

​

​

​

Good Data Science, and the decisions that flow from it, begins with accurate, relevant information - you know, data. Getting data into a form such that is usable for modeling, reporting and sound decision making is hard work. In fact, in any given Data Science project, it's fair to say that sourcing and preparing data for modeling, visualization/reporting and decisioning can be perhaps 80 percent or more of the upfront work.

​

But once data are sourced, extracted, transformed and loaded (ETLed) into useful forms, and automated processes and systems are in place to gather, model and report on them, a much larger percentage of time is spent gleaning useful insights from those data, automatically and repeatedly.

​

AnYong Analytics has significant experience in ETL and data sourcing, including:

• Database design and creation;

• Sourcing of internal and external data, both live and non-live;

• Data cleansing and quality;

• Data ETL and preparation for modeling, reporting, visualization and decision making.

Model Building & Implementation

​

​

​

​

 

 

 

The first econometric model I ever built in 1987, which was for a state regulatory filing, had four cross sections, 36 months of demand in each cross section, and took about four weeks to complete, from manual data collection all the way through final model validation and writing of documentation and testimony. It was a pooled cross-sectional times series model, using linear regression with a lagged dependent price variable, plus other explanatory variables. This one model and its related validation analytics ran in about three minutes.

​

Contrast that to the mid 2000's. I wrote a clustering and like-cluster prediction process, in which hundreds of existing-store intelligent clustering schemes were run, and then hundreds of corresponding nominal logistic models would predict, for each clustering scheme, the cluster into to which new stores would fall. This process also took about four weeks to write, including the sourcing and ETL of trade area census data, but it estimated 422 models. And this all ran in about five minutes, with predictions loaded into downstream systems, and each model's performance fully documented.

​

And today, "big data" in-memory and threaded kernel environments can read millions and even billions of rows of data partitioned across many machines, model and score scads of predictions, and render visualizations of them in dashboards and detailed reports, all in seconds. Of course, there's an upfront investment in getting relevant, quality data into these environments, but once that investment is made, the volume and speed of the modeling being performed is astounding.

​

The point is that Data Scientists have incredible resources at their disposal. So many models and tools are available to predict so many quantities and event likelihoods, but it takes experience to get to the end game. Those models are only as good as their underlying data, theoretical specifications, training and validation/holdout procedures, and proper use in decision making.

 

Let AnYong Analytics bring its experience and human insight using these resources to your model building and implementation projects. For the sake of Good Data Science.

​

Data Exploration & Visualization

​

​

​

 

 

Once data are gathered, the first thing a Data Scientist thinks is, "OK, what do we have here?" This starts with simple things like verifying that data was ETLed correctly:

• Simply looking at the data;

• Verifying data constraints (index count = index distinct count, nulls, etc);

• Verifying data dimensions, categories, hierarchies, etc.;

• Assessing data quality, missing and extreme values, additional metrics required, etc.

​

And then the fun starts when data are visualized using plots, charts, and graphs and maps of all sorts, with an eye toward modeling and decision making:

• How are data distributed statistically?

• How do data look over time, in cross section or both?

• How do data look geographically, especially as you drill down?

• How do categories of data differ, and how are they grouped or clustered?

• At which hierarchical levels are data more or less sparse?

• Which data are correlated with each other, etc.?

​

To take it up a notch, add in visualizations of predictive model results, which can be either run in other processes and stored in the data, or specified "on-the-fly", as some visualization tools allow, with little or no code.

​

Then tell the whole story in data, building dashboards and detailed reports showing when and where decisions should be made, and the results of those decisions.

​

Whether your decision is to launch a marketing campaign or a fraud investigation, to replace a HVAC component or open a store, having the visualization tools, processes and data flows to make these decisions on an ongoing, automated basis is critical. Let AnYong Analytics help you get there.

​

Targeted Campaigns & Analytics

​

​

​

​

 

 

To paraphrase a line made in a famous 1979 movie, "I love the smell of targeting in the morning." And who wouldn't? After all, you get to:

• Gather and prepare model data;

• Define the outcome (target) variable and the dimension(s) on which it is defined (customer, prospect, account, claim, transaction, loan, etc.);

• Explore and visualize input variables that explain the outcome;

• Design campaign cells for various offers, treatments, actions, etc.;

• Evaluate different models and their performance to pick a winner;

• Implement that model to score the targeted observations;

• Take action on those targeted observations;

• Evaluate and report on the outcome or success of those actions (eliciting purchases, uncovering fraudulent behavior, avoiding a loan default or retaining a loan refinance, etc.).

​

AnYong Analytics has a striking depth and breadth of experience in planning, executing and analyzing targeted campaigns from conception through conclusion.

​