Tag: data mining

Duration

14 hours (usually 2 days including breaks)

Requirements

• An understanding of Python programming.
• An understanding of Python libraries in general.

Audience

• Data analysts
• Data scientists

Overview

This instructor-led, live training (online or onsite) is aimed at data analysts and data scientists who wish to implement more advanced data analytics techniques for data mining using Python.

By the end of this training, participants will be able to:

• Understand important areas of data mining, including association rule mining, text sentiment analysis, automatic text summarization, and data anomaly detection.
• Compare and implement various strategies for solving real-world data mining problems.
• Understand and interpret the results. 

Format of the Course

• Interactive lecture and discussion.
• Lots of exercises and practice.
• Hands-on implementation in a live-lab environment.

Course Customization Options

• To request a customized training for this course, please contact us to arrange.

Course Outline

Introduction

Overview of Data Mining Concepts

Data Mining Techniques

Finding Association Rules

Matching Entities

Analyzing Networks

Analyzing the Sentiment of Text

Recognizing Named Entities

Implementing Text Summarization

Generating Topic Models

Detecting Data Anomalies

Best Practices

Summary and Conclusion

Duration

28 hours (usually 4 days including breaks)

Overview

Objective:

Delegates be able to analyse big data sets, extract patterns, choose the right variable impacting the results so that a new model is forecasted with predictive results.

Course Outline

1. Data preprocessing
   1. Data Cleaning
   2. Data integration and transformation
   3. Data reduction
   4. Discretization and concept hierarchy generation
2. Statistical inference
   1. Probability distributions, Random variables, Central limit theorem
   2. Sampling
   3. Confidence intervals
   4. Statistical Inference
   5. Hypothesis testing
3. Multivariate linear regression
   1. Specification
   2. Subset selection
   3. Estimation
   4. Validation
   5. Prediction
4. Classification methods
   1. Logistic regression
   2. Linear discriminant analysis
   3. K-nearest neighbours
   4. Naive Bayes
   5. Comparison of Classification methods
5. Neural Networks
   1. Fitting neural networks
   2. Training neural networks issues
6. Decision trees
   1. Regression trees
   2. Classification trees
   3. Trees Versus Linear Models
7. Bagging, Random Forests, Boosting
   1. Bagging
   2. Random Forests
   3. Boosting
8. Support Vector Machines and Flexible disct
   1. Maximal Margin classifier
   2. Support vector classifiers
   3. Support vector machines
   4. 2 and more classes SVM’s
   5. Relationship to logistic regression
9. Principal Components Analysis
10. Clustering
    1. K-means clustering
    2. K-medoids clustering
    3. Hierarchical clustering
    4. Density based clustering
11. Model Assesment and Selection
    1. Bias, Variance and Model complexity
    2. In-sample prediction error
    3. The Bayesian approach
    4. Cross-validation
    5. Bootstrap methods

Duration

14 hours (usually 2 days including breaks)

Requirements

Good R knowledge.

Overview

R is an open-source free programming language for statistical computing, data analysis, and graphics. R is used by a growing number of managers and data analysts inside corporations and academia. R has a wide variety of packages for data mining.

Course Outline

Sources of methods

• Artificial intelligence
• Machine learning
• Statistics
• Sources of data

Pre processing of data

• Data Import/Export
• Data Exploration and Visualization
• Dimensionality Reduction
• Dealing with missing values
• R Packages

Data mining main tasks

• Automatic or semi-automatic analysis of large quantities of data
• Extracting previously unknown interesting patterns
  • groups of data records (cluster analysis)
  • unusual records (anomaly detection)
  • dependencies (association rule mining)

Data mining

• Anomaly detection (Outlier/change/deviation detection)
• Association rule learning (Dependency modeling)
• Clustering
• Classification
• Regression
• Summarization
• Frequent Pattern Mining
• Text Mining
• Decision Trees
• Regression
• Neural Networks
• Sequence Mining
• Frequent Pattern Mining

Data dredging, data fishing, data snooping

Duration

21 hours (usually 3 days including breaks)

Requirements

Fair knowledge about relational data structures, SQL

Overview

Course can be provided with any tools, including free open-source data mining software and applications

Course Outline

Introduction

• Data mining as the analysis step of the KDD process (“Knowledge Discovery in Databases”)
• Subfield of computer science
• Discovering patterns in large data sets

Sources of methods

• Artificial intelligence
• Machine learning
• Statistics
• Database systems

What is involved?

• Database and data management aspects
• Data pre-processing
• Model and inference considerations
• Interestingness metrics
• Complexity considerations
• Post-processing of discovered structures
• Visualization
• Online updating

Data mining main tasks

• Automatic or semi-automatic analysis of large quantities of data
• Extracting previously unknown interesting patterns
  • groups of data records (cluster analysis)
  • unusual records (anomaly detection)
  • dependencies (association rule mining)

Data mining

• Anomaly detection (Outlier/change/deviation detection)
• Association rule learning (Dependency modeling)
• Clustering
• Classification
• Regression
• Summarization

Use and applications

• Able Danger
• Behavioral analytics
• Business analytics
• Cross Industry Standard Process for Data Mining
• Customer analytics
• Data mining in agriculture
• Data mining in meteorology
• Educational data mining
• Human genetic clustering
• Inference attack
• Java Data Mining
• Open-source intelligence
• Path analysis (computing)
• Reactive business intelligence

Data dredging, data fishing, data snooping

Duration

35 hours (usually 5 days including breaks)

Overview

Big data is data sets that are so voluminous and complex that traditional data processing application software are inadequate to deal with them. Big data challenges include capturing data, data storage, data analysis, search, sharing, transfer, visualization, querying, updating and information privacy.

Course Outline

Introduction to Data Science for Big Data Analytics

• Data Science Overview
• Big Data Overview
• Data Structures
• Drivers and complexities of Big Data
• Big Data ecosystem and a new approach to analytics
• Key technologies in Big Data
• Data Mining process and problems
  • Association Pattern Mining
  • Data Clustering
  • Outlier Detection
  • Data Classification

Introduction to Data Analytics lifecycle

• Discovery
• Data preparation
• Model planning
• Model building
• Presentation/Communication of results
• Operationalization
• Exercise: Case study

From this point most of the training time (80%) will be spent on examples and exercises in R and related big data technology.

Getting started with R

• Installing R and Rstudio
• Features of R language
• Objects in R
• Data in R
• Data manipulation
• Big data issues
• Exercises

Getting started with Hadoop

• Installing Hadoop
• Understanding Hadoop modes
• HDFS
• MapReduce architecture
• Hadoop related projects overview
• Writing programs in Hadoop MapReduce
• Exercises

Integrating R and Hadoop with RHadoop

• Components of RHadoop
• Installing RHadoop and connecting with Hadoop
• The architecture of RHadoop
• Hadoop streaming with R
• Data analytics problem solving with RHadoop
• Exercises

Pre-processing and preparing data

• Data preparation steps
• Feature extraction
• Data cleaning
• Data integration and transformation
• Data reduction – sampling, feature subset selection,
• Dimensionality reduction
• Discretization and binning
• Exercises and Case study

Exploratory data analytic methods in R

• Descriptive statistics
• Exploratory data analysis
• Visualization – preliminary steps
• Visualizing single variable
• Examining multiple variables
• Statistical methods for evaluation
• Hypothesis testing
• Exercises and Case study

Data Visualizations

• Basic visualizations in R
• Packages for data visualization ggplot2, lattice, plotly, lattice
• Formatting plots in R
• Advanced graphs
• Exercises

Regression (Estimating future values)

• Linear regression
• Use cases
• Model description
• Diagnostics
• Problems with linear regression
• Shrinkage methods, ridge regression, the lasso
• Generalizations and nonlinearity
• Regression splines
• Local polynomial regression
• Generalized additive models
• Regression with RHadoop
• Exercises and Case study

Classification

• The classification related problems
• Bayesian refresher
• Naïve Bayes
• Logistic regression
• K-nearest neighbors
• Decision trees algorithm
• Neural networks
• Support vector machines
• Diagnostics of classifiers
• Comparison of classification methods
• Scalable classification algorithms
• Exercises and Case study

Assessing model performance and selection

• Bias, Variance and model complexity
• Accuracy vs Interpretability
• Evaluating classifiers
• Measures of model/algorithm performance
• Hold-out method of validation
• Cross-validation
• Tuning machine learning algorithms with caret package
• Visualizing model performance with Profit ROC and Lift curves

Ensemble Methods

• Bagging
• Random Forests
• Boosting
• Gradient boosting
• Exercises and Case study

Support vector machines for classification and regression

• Maximal Margin classifiers
  • Support vector classifiers
  • Support vector machines
  • SVM’s for classification problems
  • SVM’s for regression problems
• Exercises and Case study

Identifying unknown groupings within a data set

• Feature Selection for Clustering
• Representative based algorithms: k-means, k-medoids
• Hierarchical algorithms: agglomerative and divisive methods
• Probabilistic base algorithms: EM
• Density based algorithms: DBSCAN, DENCLUE
• Cluster validation
• Advanced clustering concepts
• Clustering with RHadoop
• Exercises and Case study

Discovering connections with Link Analysis

• Link analysis concepts
• Metrics for analyzing networks
• The Pagerank algorithm
• Hyperlink-Induced Topic Search
• Link Prediction
• Exercises and Case study

Association Pattern Mining

• Frequent Pattern Mining Model
• Scalability issues in frequent pattern mining
• Brute Force algorithms
• Apriori algorithm
• The FP growth approach
• Evaluation of Candidate Rules
• Applications of Association Rules
• Validation and Testing
• Diagnostics
• Association rules with R and Hadoop
• Exercises and Case study

Constructing recommendation engines

• Understanding recommender systems
• Data mining techniques used in recommender systems
• Recommender systems with recommenderlab package
• Evaluating the recommender systems
• Recommendations with RHadoop
• Exercise: Building recommendation engine

Text analysis

• Text analysis steps
• Collecting raw text
• Bag of words
• Term Frequency –Inverse Document Frequency
• Determining Sentiments
• Exercises and Case study

Duration

14 hours (usually 2 days including breaks)

Requirements

• A basic understanding of Excel

Audience

• Data Scientists

Overview

Data mining is the process of identifying patterns in big data with data science methods such as machine learning. Using Excel as a data analytical suite, users can perform data mining and analysis.

This instructor-led, live training (online or onsite) is aimed at data scientists who wish to use Excel for data mining.

By the end of this training, participants will be able to:

• Explore data with Excel to perform data mining and analysis.
• Use Microsoft algorithms for data mining.
• Understand concepts in Excel data mining.

Format of the Course

• Interactive lecture and discussion.
• Lots of exercises and practice.
• Hands-on implementation in a live-lab environment.

Course Customization Options

• To request a customized training for this course, please contact us to arrange.

Course Outline

Introduction

Data Mining Overview

• Crisp-DM
• Algorithms for data mining
• Data mining structures and models

Preparing the Development Environment

• Installing and configuring Excel

Data Mining Algorithms and Excel to SQL Server Analysis Services

• Using Microsoft algorithms
• Working with data mining structures and models

Data Mining Add-Ins

• Working with data modeling
• Identifying influencers and categories
• Predicting with a model

Summary and Conclusion

Duration

14 hours (usually 2 days including breaks)

Requirements

• Basic knowledge of data mining process and techniques

Audience

• Data Analysts
• Data Scientists

Overview

Waikato Environment for Knowledge Analysis (Weka) is an open-source data mining visualization software. It provides a collection of machine learning algorithms for data preparation, classification, clustering, and other data mining activities.

This instructor-led, live training (online or onsite) is aimed at beginner to intermediate-level data analysts and data scientists who wish to use Weka to perform data mining tasks.

By the end of this training, participants will be able to:

• Install and configure Weka.
• Understand the Weka environment and workbench.
• Perform data mining tasks using Weka.

Format of the Course

• Interactive lecture and discussion.
• Lots of exercises and practice.
• Hands-on implementation in a live-lab environment.

Course Customization Options

• To request a customized training for this course, please contact us to arrange.

Course Outline

Introduction

• Overview of Weka
• Understanding the data mining process

Getting Started

• Installing and configuring Weka
• Understanding the Weka UI
• Setting up the environment and project
• Exploring the Weka workbench
• Loading and Exploring the dataset

Implementing Regression Models

• Understanding the different regression models
• Processing and saving processed data
• Evaluating a model using cross-validation
• Serializing and visualizing a decision tree model

Implementing Classification Models

• Understanding feature selection and data processing
• Building and evaluating classification models
• Building and visualizing a decision tree model
• Encoding text data in numeric form
• Performing classification on text data

Implementing Clustering Models

• Understanding K-means clustering
• Normalizing and visualizing data
• Performing K-means clustering
• Performing hierarchical clustering
• Performing EM clustering

Deploying a Weka Model

Troubleshooting

Summary and Next Steps

Duration

14 hours (usually 2 days including breaks)

Requirements

This course is part of the Data Scientist skill set (Domain: Analytical Techniques and Methods)

Overview

R is an open-source free programming language for statistical computing, data analysis, and graphics. R is used by a growing number of managers and data analysts inside corporations and academia. R has a wide variety of packages for data mining.

Course Outline

Introduction to Data mining and Machine Learning

• Statistical learning vs. Machine learning
• Iteration and evaluation
• Bias-Variance trade-off

Regression

• Linear regression
• Generalizations and Nonlinearity
• Exercises

Classification

• Bayesian refresher
• Naive Bayes
• Dicriminant analysis
• Logistic regression
• K-Nearest neighbors
• Support Vector Machines
• Neural networks
• Decision trees
• Exercises

Cross-validation and Resampling

• Cross-validation approaches
• Bootstrap
• Exercises

Unsupervised Learning

• K-means clustering
• Examples
• Challenges of unsupervised learning and beyond K-means

Advanced topics

• Ensemble models
• Mixed models
• Boosting
• Examples

Multidimensional reduction

• Factor Analysis
• Principal Component Analysis
• Examples

Duration

14 hours (usually 2 days including breaks)

Requirements

• Basic knowledge of data mining process and techniques

Audience

• Data Analysts
• Data Scientists

Overview

Waikato Environment for Knowledge Analysis (Weka) is an open-source data mining visualization software. It provides a collection of machine learning algorithms for data preparation, classification, clustering, and other data mining activities.

This instructor-led, live training (online or onsite) is aimed at beginner to intermediate-level data analysts and data scientists who wish to use Weka to perform data mining tasks.

By the end of this training, participants will be able to:

• Install and configure Weka.
• Understand the Weka environment and workbench.
• Perform data mining tasks using Weka.

Format of the Course

• Interactive lecture and discussion.
• Lots of exercises and practice.
• Hands-on implementation in a live-lab environment.

Course Customization Options

• To request a customized training for this course, please contact us to arrange.

Course Outline

Introduction

• Overview of Weka
• Understanding the data mining process

Getting Started

• Installing and configuring Weka
• Understanding the Weka UI
• Setting up the environment and project
• Exploring the Weka workbench
• Loading and Exploring the dataset

Implementing Regression Models

• Understanding the different regression models
• Processing and saving processed data
• Evaluating a model using cross-validation
• Serializing and visualizing a decision tree model

Implementing Classification Models

• Understanding feature selection and data processing
• Building and evaluating classification models
• Building and visualizing a decision tree model
• Encoding text data in numeric form
• Performing classification on text data

Implementing Clustering Models

• Understanding K-means clustering
• Normalizing and visualizing data
• Performing K-means clustering
• Performing hierarchical clustering
• Performing EM clustering

Deploying a Weka Model

Troubleshooting

Summary and Next Steps

Exploring the Synergy between Data Mining and AI in Advancing Machine Learning Technologies

Data mining, the process of discovering patterns and extracting valuable information from large datasets, has been a crucial component in the advancement of artificial intelligence (AI) and machine learning technologies. With the exponential growth of data generated by various industries, businesses, and individuals, the importance of data mining has become even more pronounced. The synergy between data mining and AI has led to the development of innovative machine learning algorithms and models that can analyze and interpret complex data, enabling machines to learn from experience and make intelligent decisions.

The evolution of AI and machine learning technologies has been driven by the increasing availability of data and the development of sophisticated data mining techniques. Data mining has enabled researchers and engineers to extract useful information from massive datasets, which can be used to train machine learning models. These models can then be used to make predictions, identify patterns, and solve complex problems across various domains, such as healthcare, finance, and manufacturing.

One of the key aspects of data mining that has contributed to the growth of AI and machine learning is feature extraction. Feature extraction involves identifying and selecting the most relevant variables or attributes from a dataset, which can be used to build predictive models. By selecting the most important features, data mining techniques can help reduce the complexity of machine learning models, making them more efficient and accurate.

Another significant contribution of data mining to AI and machine learning is the development of advanced algorithms for data preprocessing and transformation. Data preprocessing is an essential step in the machine learning process, as it helps clean and prepare the data for analysis. Data mining techniques, such as data cleaning, normalization, and transformation, can help improve the quality of the data and enhance the performance of machine learning models.

Moreover, data mining has played a crucial role in the development of unsupervised learning techniques, which are a subset of machine learning algorithms that do not require labeled data for training. Unsupervised learning techniques, such as clustering and dimensionality reduction, can help discover hidden patterns and structures in the data, which can be used to build more robust and accurate machine learning models.

The synergy between data mining and AI has also led to the emergence of new machine learning paradigms, such as deep learning and reinforcement learning. Deep learning, a subset of machine learning that involves training artificial neural networks to recognize patterns and make decisions, has been greatly influenced by data mining techniques. Data mining has enabled the development of advanced deep learning algorithms, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs), which can process and analyze large volumes of data with high accuracy.

Reinforcement learning, another important area of AI and machine learning, has also benefited from data mining techniques. Reinforcement learning involves training machines to make decisions based on the outcomes of their actions, with the goal of maximizing a reward signal. Data mining has helped in the development of efficient reinforcement learning algorithms, such as Q-learning and deep Q-networks (DQNs), which can learn from large amounts of data and adapt to changing environments.

In conclusion, the synergy between data mining and AI has been instrumental in redefining the landscape of machine learning technologies. The advancements in data mining techniques have enabled the development of innovative machine learning algorithms and models that can process and analyze large volumes of data with high accuracy. As the amount of data generated by various industries continues to grow, the importance of data mining in the advancement of AI and machine learning technologies will only increase. The future of AI and machine learning will be shaped by the continuous evolution of data mining techniques and their integration with cutting-edge technologies, leading to more intelligent and efficient machines that can solve complex problems and make better decisions.