6Cc526 Advanced Analytics-Statistical Analysis Answers Assessment Answers

Features 

hd3u=np.unique(hd3)

print(hd3u)

Burglary=[]

CriminalDamage=[]

Drugs=[]

FraudorForgery=[]

OtherNotifiableOffences=[]

Robbery=[]

SexualOffences=[]

TheftandHandling=[]

ViolenceAgainstthePerson=[]

for i in range(len(hd1)):

    if hd2u[0] == hd2[i]:

        Burglary.append(i)

    elif hd2u[1] == hd2[i]:

        CriminalDamage.append(i)

    elif hd2u[2] == hd2[i]:

        Drugs.append(i)

    elif hd2u[3] == hd2[i]:

        FraudorForgery.append(i)

    elif hd2u[4] == hd2[i]:

        OtherNotifiableOffences.append(i)

    elif hd2u[5] == hd2[i]:

        Robbery.append(i)

    elif hd2u[6] == hd2[i]:

        SexualOffences.append(i)

    elif hd2u[7] == hd2[i]:

        TheftandHandling.append(i)

    elif hd2u[8] == hd2[i]:

        ViolenceAgainstthePerson.append(i)

    else:

        pass

print("Burglary:n",Burglary,"n")

print("CriminalDamage:n",CriminalDamage,"n")

print("Drugs:n",Drugs,"n")

print("FraudorForgery:n",FraudorForgery,"n")

print("OtherNotifiableOffences:n",OtherNotifiableOffences,"n")

print("Robbery:n",Robbery,"n")

print("SexualOffences:n",SexualOffences,"n")

print("TheftandHandling:n",TheftandHandling,"n")

print("ViolenceAgainstthePerson:n",ViolenceAgainstthePerson,"n")

list1=file.iloc[Drugs[0]].tolist()[3:]

plt.figure(figsize=(30,10))

plt.title("Drugs Damage Comparative Analysis",fontsize=28)

for i in range(len(list1)-1):

    list3=list1+file.iloc[Drugs[i]].tolist()[3:]

plt.plot(list3,"r",label='Criminal Damage(2016-2018)')

plt.xlabel("Damage quantity")

plt.ylabel("Type of Damage")

plt.legend(loc='upper right',prop={'size': 16})

plt.grid()

Using R:

library(caret)

set.seed(7267166)

trainIndex=createDataPartition(mydata$prog, p=0.7)$Resample1

train=mydata[trainIndex, ]

test=mydata[-trainIndex, ]

print(table(mydata$prog))

print(table(train$prog))

library(e1071)   ## Classifier

NB=naiveBayes(prog~science+socst, data=train)

print(NB)

library(naivebayes)

newNB=naive_bayes(prog~sesf+science+socst,usekernel=T, data=train)

printALL(newNB)

Comparative Discussion:

In this classification issue, it was done using Naïve Bayes. This is the algorithm purely dependent upon Statistics. Python deals with it good but as R is specifically used for Statistical methods, R leads in this domain in terms of Code length and for the techniques applied and the algorithm implied in it (Saeid et al. 2017).

It is basically use to analyze the market issue. It includes the concept how a customer buys a product along with other subordinate products. If that are traceable, the owner of the market place will able to identify the associative nature of the product which are seem to be kept together for better sell (Cramer et al. 2017). It is again a statistical issue. So, here the suitable language will be R.

Using Python

import itertools"""prompt user to enter support and confidence values in percent"""

support = int(input("Please enter support value in %: "))

confidence = int(input("Please enter confidence value in %: "))

"""Compute candidate 1-itemset"""

C1 = {}

"""total number of transactions contained in the file"""

transactions = 0

D = []

T = []

with open("DataSet1.txt", "r") as f:

    for line in f:

        T = []

        transactions += 1

        for word in line.split():

            T.append(word)

            if word not in C1.keys():

                C1[word] = 1

            else:

                count = C1[word]

                C1[word] = count + 1

        D.append(T)

print ("-------------------------TEST DATASET----------------------------")

print (D)

print ("-----------------------------------------------------------------")

#prin t "--------------------CANDIDATE 1-ITEMSET------------------------- "

#print C1

#print "-----------------------------------------------------------------"

"""Compute frequent 1-itemset"""

L1 = []

for key in C1:

    if (100 * C1[key]/transactions) >= support:

        list = []

        list.append(key)

        L1.append(list)

print ("----------------------FREQUENT 1-ITEMSET-------------------------")

print (L1)

print ("-----------------------------------------------------------------")

"""apriori_gen function to compute candidate k-itemset, (Ck) , using frequent (k-1)-itemset, (Lk_1)"""

def apriori_gen(Lk_1, k):

    length = k

    Ck = []

    for list1 in Lk_1:

        for list2 in Lk_1:

            count = 0

            c = []

            if list1 != list2:

                while count < length-1:

                    if list1[count] != list2[count]:

                        break

                    else:

                        count += 1

                else:

                    if list1[length-1] < list2[length-1]:

                        for item in list1:

                            c.append(item)

                        c.append(list2[length-1])

                        if not has_infrequent_subset(c, Lk_1, k):

                            Ck.append(c)

                            c = []

    return Ck

"""function to compute 'm' element subsets of a set S"""

def findsubsets(S,m):

    return set(itertools.combinations(S, m))

"""has_infrequent_subsets function to determine if pruning is required to remove unfruitful candidates (c) using the Apriori property, with prior knowledge of frequent (k-1)-itemset (Lk_1)"""

def has_infrequent_subset(c, Lk_1, k):

    list = []

    list = findsubsets(c,k)

    for item in list:

        s = []

        for l in item:

            s.append(l)

        s.sort()

        if s not in Lk_1:

            return True

    return False

"""frequent_itemsets function to compute all frequent itemsets"""

def frequent_itemsets():

    k = 2

    Lk_1 = []

    Lk = []

    L = []

    count = 0

    transactions = 0

    for item in L1:

        Lk_1.append(item)

    while Lk_1 != []:

        Ck = []

        Lk = []

        Ck = apriori_gen(Lk_1, k-1)

        #print "-------------------------CANDIDATE %d-ITEMSET---------------------" % k

        #print "Ck: %s" % Ck

        #print "------------------------------------------------------------------"

        for c in Ck:

            count = 0

            transactions = 0

            s = set(c)

            for T in D:

                transactions += 1

                t = set(T)

                if s.issubset(t) == True:

                    count += 1

            if (100 * count/transactions) >= support:

                c.sort()

                Lk.append(c)

        Lk_1 = []

        print ("-----------------------FREQUENT %d-ITEMSET------------------------" % k)

        print (Lk)

        print ("------------------------------------------------------------------")

        for l in Lk:

            Lk_1.append(l)

        k += 1

        if Lk != []:

            L.append(Lk)

"""generate_association_rules function to mine and print all the association rules with given support and confidence value"""

def generate_association_rules():

    s = []

    r = []

    length = 0

    count = 1

    inc1 = 0

    inc2 = 0

    num = 1

    m = []

    L= frequent_itemsets()

    print ("---------------------ASSOCIATION RULES------------------")

    print ("RULES t SUPPORT t CONFIDENCE")

    print ("--------------------------------------------------------")

    for list in L:

        for l in list:

            length = len(l)

            count = 1

            while count < length:

                s = []

                r = findsubsets(l,count)

                count += 1

                for item in r:

                    inc1 = 0

                    inc2 = 0

                    s = []

                    m = []

                    for i in item:

                        s.append(i)

                    for T in D:

                        if set(s).issubset(set(T)) == True:

                            inc1 += 1

                        if set(l).issubset(set(T)) == True:

                            inc2 += 1

                    if 100*inc2/inc1 >= confidence:

                        for index in l:

                            if index not in s:

                                m.append(index)

                        print ("Rule#  %d : %s ==> %s %d %d" %(num, s, m, 100*inc2/len(D), 100*inc2/inc1))

                        num += 1  

Using R

library(arules)

class(market)

inspect(head(market, 3))

size(head(market)) # number of items in each observation

LIST(head(market, 3))

freqtItem <- eclat (market, parameter = list(supp = 0.07, maxlen = 15))

inspect(freqtItem)

itemFreq(market, topN=10, type="absolute", main="Item Frequency")

Output

Starting order_item:               32434489

Items with support >= 0.01:           10906

Remaining order_item:              29843570

Remaining orders with 2+ items:     3013325

Remaining order_item:              29662716

Item pairs:                        30622410

Item pairs with support >= 0.01:      48751

CPU times: user 11min 19s, sys: 1min 46s, total: 13min 6s

Wall time: 13min 5s

Correlation

Correlation is one of the most widely used statistical concepts it providesa definitions and intuition behind several types of correlation and illustrate how to calculate correlation using the Python pandas library (Han et al. 2017).It is a mutually connected with association. Correlation means one data is connected or related with another data with how much degree of similarity. For a business purpose, It is mandatory to maintain the clarity between the pictures of customer type and the bought items. Correlation deals with such types of topics where statistics are essential.

Using Python

import pandas as pd

path =

mpg_data = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/auto-mpg/auto-mpg.data', delim_whitespace=True, header=None,names = ['mpg', 'cylinder', 'displacemnt','horsepwr','weight', 'acceleration', 'model_year', 'origin', 'name'],

            na_values='?')

mpg_data['mpg'].corr(mpg_data['weight'])

# pairwise correlation

mpg_data.drop(['model_year', 'origin'], axis=1).corr(method='sperman')

# plot correlated values

plt.rcParams['figure.figsize'] = [16, 6]

fig, ax = plt.subplots(nrws=1, ncls=3)

ax=ax.flatten()

cols = ['weight', 'horsepower', 'acceleration']

colors=['#415052', '#f33234', '#243AC5', '#2442B5']

j=0

for i in ax:

    if j==0:

        i.set_ylabel('MPG')

    i.scatter(mpg_data[cols[j]], mpg_data['mpg'],  alpha=0.5, color=colors[j])

    i.set_xlabel(cols[j])

    i.set_title('Pearson: %s'%mpg_data.corr().loc[cols[j]]['mpg'].round(2)+' Spearman: %s'%mpg_data.corr(method='spearman').loc[cols[j]]['mpg'].round(2))

    j+=

plt.show()

Using R

my_data <- read.csv(file.choose())

my_data <- mtcars

head(my_data, 6)

library("ggpubr")

ggscatter(my_data, x = "mpg", y = "wt",

          add = "reg.line", conf.int = TRUE,

          cor.coef = TRUE, cor.method = "pearson",

          xlab = "Miles/(US) gallon", ylab = "Weight (1000 lbs)")

library("ggpubr")

ggqqplot(my_data$mpg, ylab = "MPG")

ggqqplot(my_data$wt, ylab = "WT")

res <- cor.test(my_data$wt, my_data$mpg,

                    method = "pearson")

print(res)

Output

In the simulation of correlation and its output, we can see likewise the structure and methods involved in association, R leads because of the richer library. Statistical calculation are the lengthiest in programming because it requires thorough analysis. As R is richer in statistical methods and library, a few line of codes are required to execute a whole application. So execution time will be lesser.

Regression

It is the technique or algorithm to relate the input with target data set. Here the example is shown regarding linear model. Let we see the example.

Using Python

list1=file.iloc[CriminalDamage[0]].tolist()[3:]

plt.figure(figsize=(30,10))

plt.title("Criminal Damage Comparative Analysis",fontsize=28)

for i in range(len(list1)-1):

list4=list1+file.iloc[CriminalDamage[i]].tolist()[3:]

plt.plot(list4,"c",label='Criminal Damage(2016-2018)')

plt.xlabel("Damage quantity")

plt.ylabel("Type of Damage")

plt.legend(loc='upper right',prop={'size': 16})

plt.grid()

list1=file.iloc[FraudorForgery[0]].tolist()[3:]

plt.figure(figsize=(30,10))

plt.title("Fraudor Forgery Comparative Analysis",fontsize=28)

for i in range(len(list1)-1):

    list5=list1+file.iloc[CriminalDamage[i]].tolist()[3:]

plt.plot(list5,"c",label='Fraudor Forgery(2016-2018)')

plt.xlabel("Type of Frauder")

plt.ylabel("Frauder Count")

plt.legend(loc='upper right',prop={'size': 16})

plt.grid()

Using R

library(e1071)

par(mfrow=c(1, 2))  # divide graph area in 2 columns

plot(density(cars$speed), main="Density Plot: Speed", ylab="Frequency", sub=paste("Skewness:", round(e1071::skewness(cars$speed), 2)))  # density plot for 'speed'

polygon(density(cars$speed), col="red")

plot(density(cars$dist), main="Density Plot: Distance", ylab="Frequency", sub=paste("Skewness:", round(e1071::skewness(cars$dist), 2)))  # density plot for 'dist'

polygon(density(cars$dist), col="red")

par(mfrow=c(1, 2))  # divide graph area in 2 columns

boxplot(cars$speed, main="Speed", sub=paste("Outlier rows: ", boxplot.stats(cars$speed)$out))  # box plot for 'speed'

boxplot(cars$dist, main="Distance", sub=paste("Outlier rows: ", boxplot.stats(cars$dist)$out))

Output

Comparative Discussion:

In this operation, preferably statistic is not used or been barely used. So, we can use python as well as R in this domain of work.

Clustering

Clustering is an unsupervised learning where we have to segmentize data which are not known to us. It is basically used for web filtering or stock market analysis or such type of work where the data type may be unknown to us.

Using Python

-------------------------------Importing Packages-------------------------------------

import pandas as pd

import numpy as np

import matplotlib.pyplot as plt

get_ipython().run_line_magic('matplotlib', 'inline')

#-------------------------------Creating Data Frame-----------------------------------

df = pd.DataFrame({

    'x': [12, 20, 28, 18, 29, 33, 24, 45, 45, 52, 51, 52, 55, 53, 55, 61, 64, 69, 72,51,12,22,36,45,65,15,19,31,81,26],

    'y': [39, 36, 30, 52, 54, 46, 55, 59, 63, 70, 66, 63, 58, 23, 14, 8, 19, 7, 24,33,43,21,8,34,54,12,11,19,38,49]

})

plt.scatter(df['x'],df['y'])

--------------------------Creating & Assigning Centroid------------------------------

np.random.seed(200)

k = 3

 centroids[i] = [x, y]

centroids = {

 i+1: [np.random.randint(0, 80), np.random.randint(0, 80)]

 for i in range(k)

}

print(centroids)

list1=[]

list1=centroids[1]

list2=centroids[2]

list3=centroids[3]

plt.figure()

a=list1[0]

b=list1[1]

plt.plot(a,b,'*r',0.6)

plt.figure()

a=list2[0]

b=list2[1]

plt.plot(a,b,'*m',0.6)

plt.figure()

a=list3[0]

b=list3[1]

plt.plot(a,b,'*c',0.6)

#-------------------------------Plotting of points(seed)-----------------------------    

fig = plt.figure(figsize=(5, 5))

plt.scatter(df['x'], df['y'], color='k')

colmap = {1: 'r', 2: 'g', 3: 'b'}

for i in centroids.keys():

    plt.scatter(*centroids[i], color=colmap[i])

plt.xlim(0, 80)

plt.ylim(0, 80)

plt.show()

#------------------------------ Centroid Assignment Stage(Euclidian Distance) ---------------------------

def assignment(df, centroids):

    for i in centroids.keys():

        # sqrt((x1 - x2)^2 - (y1 - y2)^2)

        df['distance_from_{}'.format(i)] = (

            np.sqrt(

                (df['x'] - centroids[i][0]) ** 2

                + (df['y'] - centroids[i][1]) ** 2

    centroid_distance_cols = ['distance_from_{}'.format(i) for i in centroids.keys()]

    df['closest'] = df.loc[:, centroid_distance_cols].idxmin(axis=1)

    df['closest'] = df['closest'].map(lambda x: int(x.lstrip('distance_from_')))

    df['color'] = df['closest'].map(lambda x: colmap[x])

    return dfds with Centroid-------------------------

fig = plt.figure(figsize=(5, 5))

plt.scatter(df['x'], df['y'], color=df['color'], alpha=0.5, edgecolor='k')

for i in centroids.keys():

    plt.scatter(*centroids[i], color=colmap[i])

plt.xlim(0, 80)

plt.ylim(0, 80)

plt.show()---

def update(k):

    for i in centroids.keys():

        centroids[i][0] = np.mean(df[df['closest'] == i]['x'])

        centroids[i][1] = np.mean(df[df['closest'] == i]['y'])

    return k

centroids = update(centroids)

#--------------------------------Plot updated seeds in cluster form--------------------    

fig = plt.figure(figsize=(5, 5))

ax = plt.axes()

plt.scatter(df['x'], df['y'], color=df['color'], alpha=0.5, edgecolor='k')

for i in centroids.keys():

    plt.scatter(*centroids[i], color=colmap[i])

plt.xlim(0, 80)

plt.ylim(0, 80)

for i in old_centroids.keys():

    old_x = old_centroids[i][0]

    old_y = old_centroids[i][1]

    dx = (centroids[i][0] - old_centroids[i][0]) * 0.75

    dy = (centroids[i][1] - old_centroids[i][1]) * 0.75

    ax.arrow(old_x, old_y, dx, dy, head_width=2, head_length=3, fc=colmap[i], ec=colmap[i])

plt.show()

Using R

library(VIM)

data14$Date.Time <- mdy_hms(data14$Date.Time)

data14$Year <- factor(year(data14$Date.Time))

data14$Month <- factor(month(data14$Date.Time))

data14$Day <- factor(day(data14$Date.Time))

data14$Weekday <- factor(wday(data14$Date.Time))

data14$Hour <- factor(hour(data14$Date.Time))

data14$Minute <- factor(minute(data14$Date.Time))

data14$Second <- factor(second(data14$Date.Time))

set.seed(10)

clusters <- kmeans(data14[,2:3], 5)

library(ggmap)

NYCMap <- get_map("New York", zoom = 10)

ggmap(NYCMap) + geom_point(aes(x = Lon[], y = Lat[], colour = as.factor(Borough)),data = data14) +

  ggtitle("NYC Boroughs using KMean")

Comparative Discussion:

In this operation, preferably statistic is not used or been barely used. So, we can use python as well as R in this domain of work. But if there is a necessity of using the statistical concept in it, we should go for R rather by using Python.

Anomaly Detection

This is type of outlier detection where in the data set , may be known or unknown to us, some parameters are changed suspiciously. This is one of the important aspect of our consideration because those parameters are need to know to catch the entire simulation process and for future aspects.

Using Python

import matplotlib.pyplot as plt

import numpy as np

%matplotlib inline

from numpy import genfromtxt

from scipy.stats import multivariate_normal

from sklearn.metrics import f1_score

def selectThresholdByCV(probs,gt):

    best_epsilon = 0

    best_f1 = 0

    f = 0

    stepsize = (max(probs) - min(probs)) / 1000;

    epsilons = np.arange(min(probs),max(probs),stepsize)

    for epsilon in np.nditer(epsilons):

        predictions = (probs < epsilon)

        f = f1_score(gt, predictions, average = "binary")

        if f > best_f1:

            best_f1 = f

            best_epsilon = epsilon

    return best_f1, best_epsilon

tr_data = read_dataset('tr_server_data.csv')

cv_data = read_dataset('cv_server_data.csv')

gt_data = read_dataset('gt_server_data.csv')

n_training_samples = tr_data.shape[0]

n_dim = tr_data.shape[1]

plt.figure()

plt.xlabel("Latency (ms)")

plt.ylabel("Throughput (mb/s)")

plt.plot(tr_data[:,0],tr_data[:,1],"bx")

plt.show()

mu, sigma = estimateGaussian(tr_data)

p = multivariateGaussian(tr_data,mu,sigma)

p_cv = multivariateGaussian(cv_data,mu,sigma)

fscore, ep = selectThresholdByCV(p_cv,gt_data)

outliers = np.asarray(np.where(p < ep))

plt.figure()

plt.xlabel("Latency (ms)")

plt.ylabel("Throughput (mb/s)")

plt.plot(tr_data[:,0],tr_data[:,1],"bx") plt.plot(tr_data[outliers,0],tr_data[outliers,1],"ro")

plt.show()

Comparative Discussion:

In anomaly detection, statistical analysis is essentially required. In the example program the simulation of bitcoin is specie which one of the most hot topic on which we can concentrate because there are many parameters available for which the demand of bitcoin goes ups and down.

Reference list

Gruden, G., Giunti, S., Barutta, F., Chaturvedi, N., Witte, D.R., Tricarico, M., Fuller, J.H., Perin, P.C. and Bruno, G., 2012. QTc interval prolongation is independently associated with severe hypoglycemic attacks in type 1 diabetes from the EURODIAB IDDM complications study. Diabetes care, 35(1), pp.125-127.

Hunter, J.D., 2007. Matplotlib: A 2D graphics environment. Computing in science & engineering, 9(3), pp.90-95.

Cer?ak, M., 2012. A comparison of decision tree classifiers for automatic diagnosis of speech recognition errors. Computing and Informatics, 29(3), pp.489-501.

Längkvist, M., Karlsson, L. and Loutfi, A., 2014. A review of unsupervised feature learning and deep learning for time-series modeling. Pattern Recognition Letters, 42, pp.11-24.

Prakash, M. and Singaravel, G., 2015. An approach for prevention of privacy breach and information leakage in sensitive data mining. Computers & Electrical Engineering, 45, pp.134-140.

Mohammad Saeid Mahdavinejad, Mohammadreza Rezvan, Mohammadamin Barekatain, Peyman Adibi, Payam Barnaghi, Amit P. Sheth, "Machine learning for Internet of Things data analysis: A survey" in Digital Communications and Networks, Elsevier, 2017.

Kwon, O. and Sim, J.M., 2013. Effects of data set features on the performances of classification algorithms. Expert Systems with Applications, 40(5), pp.1847-1857.

Han, R., John, L.K. and Zhan, J., 2018. Benchmarking big data systems: A review. IEEE Transactions on Services Computing, 11(3), pp.580-597.

Cramer, S., Kampouridis, M., Freitas, A.A. and Alexandridis, A.K., 2017. An extensive evaluation of seven machine learning methods for rainfall prediction in weather derivatives. Expert Systems with Applications, 85, pp.169-181.

David, S.K., Saeb, A.T. and Al Rubeaan, K., 2013. Comparative analysis of data mining tools and classification techniques using weka in medical bioinformatics. Computer Engineering and Intelligent Systems, 4(13), pp.28-38.Show Context CrossRef Google Scholar

Show Context CrossRef Google Scholar

Show Context View Article Full Text: PDF (284KB) Google Scholar

Salaken, S.M., Khosravi, A., Nguyen, T. and Nahavandi, S., 2017. Extreme learning machine based transfer learning algorithms: A survey. Neurocomputing, 267, pp.516-524.

Christensen, T.F., Lewinsky, I., Kristensen, L.E., Randlov, J., Poulsen, J.U., Eldrup, E., Pater, C., Hejlesen, O.K. and Struijk, J.J., 2007, September. QT Interval prolongation during rapid fall in blood glucose in type I diabetes. In Computers in Cardiology, 2007 (pp. 345-348). IEEE.

Christensen, T.F., Tarnow, L., Randløv, J., Kristensen, L.E., Struijk, J.J., Eldrup, E. and Hejlesen, O.K., 2010. QT interval prolongation during spontaneous episodes of hypoglycaemia in type 1 diabetes: the impact of heart rate correction. Diabetologia, 53(9), pp.2036-2041.

Kumari, V.A. and Chitra, R., 2013. Classification of diabetes disease using support vector machine. International Journal of Engineering Research and Applications, 3(2), pp.1797-1801.

Antony, P.J., Manujesh, P. and Jnanesh, N.A., 2016, May. Data mining and machine learning approaches on engineering materials—a review. In Recent Trends in Electronics, Information & Communication Technology (RTEICT), IEEE International Conference on (pp. 69-73). IEEE.

Nelson, B. and Olovsson, T., 2016, December. Security and privacy for big data: A systematic literature review. In Big Data (Big Data), 2016 IEEE International Conference on (pp. 3693-3702). IEEE.

Dincer, C., Akpolat, G. and Zeydan, E., 2017, May. Security issues of big data applications served by mobile operators. In Signal Processing and Communications Applications Conference (SIU), 2017 25th (pp. 1-4). IEEE.

Volk, M., Bosse, S. and Turowski, K., 2017, July. Providing Clarity on Big Data Technologies: A Structured Literature Review. In Business Informatics (CBI), 2017 IEEE 19th Conference on (Vol. 1, pp. 388-397). IEEE.

Prakash, M., Padmapriy, G. and Kumar, M.V., 2018, April. A Review on Machine Learning Big Data using R. In 2018 Second International Conference on Inventive Communication and Computational Technologies (ICICCT) (pp. 1873-1877). IEEE.