1基本介紹1.1簡介RHadoop是運行R語言的Hadoop分布式計算平臺的簡稱。要認(rèn)識Rhadoop首先我們分別來認(rèn)識R語言和Hadoop。Hadoop已經(jīng)成名好多年了，它是Apache軟件基金會旗下的一個JAVA開源分布式計算平臺，現(xiàn)已被各大互聯(lián)網(wǎng)企業(yè)（包括Yahoo和Facebook等）用于大規(guī)模數(shù)據(jù)分布式存儲與分布式計算。近年來隨著云計算、大數(shù)據(jù)處理、數(shù)據(jù)挖掘等概念和應(yīng)用越來越火，Hadoop更是名聲大噪，各大企業(yè)對熟悉Hadoop體系架構(gòu)和性能優(yōu)化的人才需求也相當(dāng)旺盛，人才缺口也相對加大。R語言可能對大多數(shù)人來說比較陌生，但是做數(shù)理統(tǒng)計、數(shù)據(jù)分析的人不會不知道R。R是一門主要運用于統(tǒng)計分析與繪圖的編程語言和操作工具。和Matlab類似，R是一門高級腳本化編程語言，直接在命令行輸入指令即可運行，無需編譯鏈接。它可以進行高效的向量化運算，效率遠高于傳統(tǒng)的循環(huán)運算。另外，R是開源的，有大批杰出的工程師和程序員為R開發(fā)了許多好用的函數(shù)庫和圖形化工具，尤其在數(shù)理統(tǒng)計方面有大量成熟的開源的程序包。所以R是做數(shù)據(jù)挖掘、數(shù)據(jù)分析的廉價高效的方案。1.2功能Rhadoop是將R的強大統(tǒng)計分析能力和hadoop的大數(shù)據(jù)處理能力相結(jié)合，可由以下幾項功能組成。R內(nèi)置多種統(tǒng)計學(xué)及數(shù)字分析功能R的另一強項是繪圖功能，制圖具有印刷的素質(zhì)，也可加入數(shù)學(xué)符號Hadoop分布式文件系統(tǒng)，高吞吐量的數(shù)據(jù)級別達到TB、PB甚至EBMapreduce高效并行計算1.3常用算法包Rhadoop中的數(shù)據(jù)挖掘算法均來自R語言（或者說R軟件），常用算法包如下：1.3.1聚類? 常用的包：fpc，cluster，pvclust，mclust?基于劃分的方法?基于劃分的方法?基于層次的方法?基于模型的方法?基于密度的方法?基于畫圖的方法基于驗證的方法：cluster.stats1.3.2分類.常用的包：rpart,party,randomForest,rpartOrdinal,tree,marginTree,maptree,survival.決策樹：rpart,ctree.隨機森林：cforest,randomForest.回歸，Logistic回歸，Poisson回歸：glm,predict,residuals.生存分析：survfit,survdiff,coxph1.3.3關(guān)聯(lián)規(guī)則與頻繁項集.常用的包：arules:支持挖掘頻繁項集，最大頻繁項集，頻繁閉項目集和關(guān)聯(lián)規(guī)則DRM:回歸和分類數(shù)據(jù)的重復(fù)關(guān)聯(lián)模型.APRIORI算法，廣度RST算法：apriori,drm.ECLAT算法：采用等價類，RST深度搜索和集合的交集：eclat1.3.4序列模式.常用的包：arulesSequences.SPADE算法：cSPADE1.3.5時間序列.常用的包：timsac.時間序列構(gòu)建函數(shù)：ts.成分分解：decomp,decompose,stl,tsr1.3.6人工神經(jīng)網(wǎng)絡(luò).nnet1.3.7支持向量機e1071

常用的包：BaseR,nlme方差分析：aov,anova密度分析：density假設(shè)檢驗：t.test,prop.test,anova,aov線性混合模型：lme主成分分析和因子分析：princomp1.3.9接入數(shù)據(jù)挖掘軟件Weka算法包RWeka:通過這個接口，可以在R中使用Weka的所有算法。2系統(tǒng)架構(gòu)RHadoop是R環(huán)境+hadoop環(huán)境，是實際上是在hadoop集群中安裝了R環(huán)境，通過R的三個包rmr、rhdfs、rhbase（分別是對應(yīng)Hadoop系統(tǒng)架構(gòu)中的，MapReduce,HDFS,HBase三個部分），實現(xiàn)R對hadoop各個組件的調(diào)用，從而將R與hadoop平臺結(jié)合，從而充分利用了R和hadoop各自的優(yōu)勢°Hadoop主要用來存儲海量數(shù)據(jù)，R語言完成MapReduce算法，用來替代Java的MapReduce實現(xiàn)，將R豐富的算法模型賦予分布式并行計算能力。3適用場景Hadoop有海量數(shù)據(jù)存儲能力和高效并行計算框架，但缺乏數(shù)據(jù)統(tǒng)計和建模能力，R語言可以為hadoop注入統(tǒng)計血脈，增強hadoop集群的數(shù)據(jù)統(tǒng)計分析和建模能力，R+Hadoop將實現(xiàn)高效的海量數(shù)據(jù)分析與挖掘技術(shù)。雖然hadoop軟件家族中的mahout也可以實現(xiàn)機器學(xué)習(xí)算法的并行計算，但mahout的算法模型還是較少和不夠成熟，并且本身是一個算法庫，它缺少R語言（或者說R軟件）的數(shù)據(jù)統(tǒng)計分析以及繪圖能力，因而Rhadoop相比mahout有諸多優(yōu)勢。4優(yōu)缺點分析4.1優(yōu)缺點概覽;數(shù)據(jù)計算能力說明優(yōu)劣勢利用hadoop;數(shù)據(jù)計算能力說明優(yōu)劣勢利用hadoop平臺的優(yōu)秀的大數(shù)據(jù)處理和分布式并行計算能力優(yōu)勢數(shù)據(jù)存儲 利用hadoop平臺的分布式文件系統(tǒng)存儲 優(yōu)勢數(shù)據(jù)處理豐富的數(shù)據(jù)處理函數(shù)和擴展包，靈活的數(shù)據(jù)處理能力優(yōu)勢算法模型提供大量的成熟算法優(yōu)勢自動建模不提供靈活的參數(shù)設(shè)置等功能劣勢可視化雖然提供繪圖功能，但命令式編程，沒有可視化UI。劣勢經(jīng)濟性開源優(yōu)勢擴展性R語言本身擴展性較強，可以添加額外的包優(yōu)勢技術(shù)支持有官方開源社區(qū)支持優(yōu)勢學(xué)習(xí)成本涉及知識點較廣、如需掌握java、linux、hadoop、R等劣勢發(fā)展空間R和Hadoop的結(jié)合，發(fā)展空間廣闊優(yōu)勢4?2主要優(yōu)缺點說明所以如果讓能R運行在Hadoop平臺上可以說是如虎添翼，可以讓數(shù)據(jù)分析人員，即能用到R豐富的程序包，又能借助Hadoop分布式運算、分布式存儲的優(yōu)勢做海量數(shù)據(jù)的處理。但由于R+Hadoop還不夠成熟，企業(yè)級應(yīng)用依然缺乏成功案例，所以當(dāng)實施R+Hadoop的應(yīng)用時，還會碰到非常多的問題。5擴展性Hadoop的高擴展性和R語言的高擴展性決定rhadoop的高擴展性6實際案例6.1簡單示例(WordCount的例子)使用rmr包和rhdfs包做個WordCount的試驗。首先要啟動Hadoop。system('start-all.sh')system('hadoopdfsadmin-safemodeleave')加載R包。require(rmr)require(rhdfs)rhdfs包會建議做一個初始化測試。hdfs.init()編寫wordcount函數(shù)。wordcount<-function(input,output二NULL,split='[[:punct:][:space:]]+'){mapreduce(input二input,output=output,map二function(k,v)(v2=unlist(strsplit(x=v,split二split))v3=v2[v2!='']lapply(v3,function(w){keyval(w,1)})},reduce=function(k,vv)(keyval(k,sum(unlist(vv)))})}直接輸入文本，轉(zhuǎn)成HDFS文件，計數(shù)。lines<-c('thisisjustajoke,','andthatisjustanotherjoke,','wealllikethemverymuch,','becausetheyaresofunny.')lines_dfs<-to.dfs(lines)wc1<-wordcount(input=lines_dfs)data.frame(do.call(rbind,from.dfs(wc1)))keyvalTOC\o"1-5"\h\za1is2so1we1all1and1are1joke2just2like1much1that1them1they1this1very1funny1another1because16.2R實現(xiàn)MapReduce的協(xié)同過濾算法基于物品推薦的協(xié)同過濾算法介紹R本地程序?qū)崿F(xiàn)R基于Hadoop分步式程序?qū)崿F(xiàn)每一章節(jié)，都會分為”文字說明部分”和”代碼部分”，保持文字說明與代碼的連貫性。6.2.1基于物品推薦的協(xié)同過濾算法介紹文字說明部分：越來越多的互聯(lián)網(wǎng)應(yīng)用，都開始使用推薦算法（協(xié)同過濾算法）。根據(jù)用戶活躍度和物品流行度，可以分為”基于用戶的協(xié)同過濾算法”和”基于物品的協(xié)同過濾算法”。基于用戶的協(xié)同過濾算法，是給用戶推薦和他興趣相似的其他用戶喜歡的物品。基于物品的協(xié)同過濾算法，是給用戶推薦和他之前喜歡的物品相似的物品。基于物品的協(xié)同過濾算法，是目前廣泛使用的一種推薦算法，像Netflix,YouTube,Amazon等。算法主要分為兩步：計算物品之間的相似度根據(jù)物品的相似度和用戶的歷史行為給用戶生成推薦列表有關(guān)算法的細節(jié)請參考：”MahoutInAction”和”推薦系統(tǒng)實踐”兩本書。為開發(fā)方便，我們選擇一組很小的測試數(shù)據(jù)集。測試數(shù)據(jù)，來自于"MahoutInAction”P49原第8行，3,101,2.5改為3,101,2.0每行3個字段，依次是用戶ID,物品ID,對物品的評分代碼部分：在服務(wù)上創(chuàng)建測試數(shù)據(jù)文件small.csv?pwd/root/R?vismall.csv1,101,5.01,102,3.01,103,2.52,101,2.02,102,2.52,103,5.03,101,2.03,104,4.03,105,4.53,107,5.04,101,5.04,103,3.04,104,4.54,106,4.05,101,4.05,102,3.05,103,2.05,104,4.05,105,3.55,106,4.0~lssmall.csv6.2.2R本地程序?qū)崿F(xiàn)首先，通過R語言實現(xiàn)基于物品的協(xié)同過濾算法，為和RHadoop實現(xiàn)進行對比。這里我使用”MahoutInActic書里，第一章第六節(jié)介紹的分步式基于物品的協(xié)同過濾算法進行實現(xiàn)。Chapter6:Distributingrecommendationcomputations算法的思想：建立物品的同現(xiàn)矩陣建立用戶對物品的評分矩陣矩陣計算推薦結(jié)果文字說明部分：建立物品的同現(xiàn)矩陣按用戶分組，找到每個用戶所選的物品，單獨出現(xiàn)計數(shù)，及兩兩一組計數(shù)。例如：用戶ID為3的用戶，分別給101,104,105,107，這4個物品打分。(101,101),(104,104),(105,105),(107,107)，單獨出現(xiàn)計算各加1。(101,104),(101,105),(101,107),(104,105),(104,107),(105,107)，兩個一組計數(shù)各加1。把所有用戶的計算結(jié)果求和，生成一個三角矩陣，再補全三角矩陣，就建立了物品的同現(xiàn)矩陣。如下面矩陣所示：[102] [103] [104] [105] [106] [107]

[102]3332110[103]4343120[104]4234221[105]2112211[106]2122120[107]1001101建立用戶對物品的評分矩陣按用戶分組，找到每個用戶所選的物品及評分例如：用戶ID為3的用戶，分別給(3,101,2.0),(3,104,4.0),(3,105,4.5),(3,107,5.0)，這4個物品打分。找到物品評分(3,101,2.0),(3,104,4.0),(3,105,4.5),(3,107,5.0)建立用戶對物品的評分矩陣U3TOC\o"1-5"\h\z2.00.00.04.04.50.05.0矩陣計算推薦結(jié)果同現(xiàn)矩陣*評分矩陣=推薦結(jié)果代碼部分:

#引用plyr包library(plyr)#讀取數(shù)據(jù)集train<-read.csv(file="small.csv”,header=FALSE)names(train)<-c("user”,"item”,"pref")>trainuseritempref11015.011023.011032.521012.021022.521035.021042.031012.031044.031054.531075.041015.041033.041044.541064.051014.051023.051032.051044.051053.551064.0#計算用戶列表usersUnique<-function(){users<-unique(train$user)users[order(users)]}#計算商品列表方法itemsUnique<-function(){items<-unique(train$item)items[order(items)]}

#用戶列表users<-usersUnique()users[1]12345#商品列表items<-itemsUnique()items[1]101102103104105106107#建立商品列表索引index<-function(x)which(items%in%x)data<-ddply(train,.(user,item,pref),summarize,idx=index(item))>datauseritemprefidxTOC\o"1-5"\h\z1 101 5.0 11 102 3.0 21 103 2.5 32 101 2.0 12 102 2.5 22 103 5.0 32 104 2.0 43 101 2.0 13 104 4.0 43 105 4.5 53 107 5.0 74 101 5.0 14 103 3.0 34 104 4.5 44 106 4.0 65 101 4.0 15 102 3.0 25 103 2.0 35 104 4.0 45 105 3.5 55 106 4.0 6#同現(xiàn)矩陣cooccurrence<-function(data){n<-length(items)co<-matrix(rep(0,n*n),nrow=n)for(uinusers){idx<-index(data$item[which(data$user==u)])

m<-merge(idx,idx)for(iin1:nrow(m)){co[m$x[i],m$y[i]]=co[m$x[i],m$y[i]]+1}}return(co)}#推薦算法recommend<-function(udata=udata,co=coMatrix,num=0){n<-length(items)#allofprefpref<-rep(0,n)pref[udata$idx]<-udata$pref#用戶評分矩陣userx<-matrix(pref,nrow=n)#同現(xiàn)矩陣*評分矩陣r<-co%*%userx#推薦結(jié)果排序r[udata$idx]<-0idx<-order(r,decreasing=TRUE)topn<-data.frame(user=rep(udata$user[1],length(idx)),item=items[idx],val=r[idx])topn<-topn[which(topn$val>0),]#推薦結(jié)果取前num個if(num>0){topn<-head(topn,num)}#返回結(jié)果return(topn)}#生成同現(xiàn)矩陣co<-cooccurrence(data)>co[,1][,2][,3][,4][,5][,6][,7][1,]5344221[2,]3332110

[3,]4343120[4,]4234221[5,]2112211[6,]2122120[7,]1001101#計算推薦結(jié)果recommendation<-data.frame()for(iin1:length(users)){udata<-data[which(data$user==users[i]),]recommendation<-rbind(recommendation,recommend(udata,co,0)):re_useritemval110433.5110618.0110515.511075.0210620.5210515.521074.0310324.5310218.5310616.5410237.0410526.041079.5510711.56.2.3R基于Hadoop分步式程序?qū)崿F(xiàn)R語言實現(xiàn)的MapReduce算法，可以基于R的數(shù)據(jù)對象實現(xiàn)，不必如JAVA一樣使用文本存儲。算法思想同上面R語言實現(xiàn)思想，略有復(fù)雜。算法的思想：建立物品的同現(xiàn)矩陣按用戶分組，得到所有物品出現(xiàn)的組合列表。對物品組合列表進行計數(shù)，建立物品的同現(xiàn)矩陣建立用戶對物品的評分矩陣合并同現(xiàn)矩陣和評分矩陣

計算推薦結(jié)果列表按輸入格式得到推薦評分列表通過MapReduce實現(xiàn)時，所有操作都要使用Map和Reduce的任務(wù)完成，程序?qū)崿F(xiàn)過程略有變化。文字說明部分:建立物品的同現(xiàn)矩陣1）按用戶分組，得到所有物品出現(xiàn)的組合列表。key:物品列表向量val:物品組合向量$key[1]101101101101101101101101101101101101101101101102102102102[20]102102102103103103103103103103103103103103104104104104104[39]104104104104104104104105105105105106106106106107107107107

[58]101101101101101101102102102102102102103103103103103103104[77]104104104104104105105105105105105106106106106106106$val[1]101102103101102103104101104105107101103104106101102103101[20]102103104101102103101102103104101103104106101102103104101[39]104105107101103104106101104105107101103104106101104105107[58]101102103104105106101102103104105106101102103104105106101[77]1021031041051061011021031041051061011021031041051062)對物品組合列表進行計數(shù)，建立物品的同現(xiàn)矩陣key:物品列表向量val:同現(xiàn)矩陣的數(shù)據(jù)框值(item,item,Freq)矩陣格式，要與”2.建立用戶對物品的評分矩陣”的格式一致，把異構(gòu)的兩種數(shù)據(jù)源，合并為同一種數(shù)據(jù)格式，為”3.合并同現(xiàn)矩陣和評分矩陣”做數(shù)據(jù)基礎(chǔ)。$key[1]101101101101101101101102102102102102102103103103103103103[20]104104104104104104104105105105105105105105106106106106106[39]106107107107107$valkvfreq11011015210110233101103441011044510110526101106271011071810210139102102310102103311102104212102105113102106114103101415103102316103103417103104318103105119103106220104101421104102210410331041044104105210410621041071105101210510211051031105104210510521051061331051071341061012351061021361061032371061042381061051391061062401071011411071041421071051431071071建立用戶對物品的評分矩陣key:物品列表val:用戶對物品打分矩陣矩陣格式，要與”2）對物品組合列表進行計數(shù)，建立物品的同現(xiàn)矩陣”的格式一致，把異構(gòu)的兩種數(shù)據(jù)源，合并為同一種數(shù)據(jù)格式，為”3.合并同現(xiàn)矩陣和評分矩陣”做數(shù)據(jù)基礎(chǔ)$key[1]101101101101101102102102103103103103104104104104105105106[20]106107$valitemuserpref110115.0210122.0310132.0410145.0510154.0610213.0710222.5810253.0910312.510325.010343.010352.010422.010434.010444.510454.010534.510553.510644.010654.010735.0合并同現(xiàn)矩陣和評分矩陣這一步操作是MapReduce比較特殊的，因為數(shù)據(jù)源是兩個異構(gòu)數(shù)據(jù)源，進行MapReduce的操作。在之前，我們已經(jīng)把兩種格式合并為一樣的。使用equijoin這個rmr2包的函數(shù)，進行矩陣合并。key:NULLval:合并的數(shù)據(jù)框$keyNULL$valk.lv.lfreq.litem.ruser.rpref.r1103101410312.52103102310312.53103103410312.54103104310312.55103105110312.56103106210312.57103101410325.08103102310325.09103103410325.010103104310325.011103105110325.012103106210325.013103101410343.0計算推薦結(jié)果列表把第三步中的矩陣，進行合并計算，得到推薦結(jié)果列表key:物品列表val:推薦結(jié)果數(shù)據(jù)框

$key[1]101101101101101101101101101101101101101101101101101101101101101101101101101102[19]101101101101101101101101101101[37]102102102102102102102102102102102102102102102102102103[55]103103103103103103103103103103103103103103103103103103[73]103103103103103104104104104104104104104104104104104104[91]104104104104104104104104104104104104104104104105105105[109]105105105105105105105105105105105106106106106106106106[127]106106106106106107107107107$valk.lv.luser.rv1101101125.02101101210.03101101310.04101101425.05101101520.06101102115.0710110226.0810110236.09101102415.0101102512.0101103120.010110328.010110338.0101103420.0101103516.0101104120.010110428.010110438.0....按輸入格式得到推薦評分列表對推薦結(jié)果列表，進行排序處理，輸出排序后的推薦結(jié)果。key:用戶IDval:推薦結(jié)果數(shù)據(jù)框$key[1]11111112222222333333344444445555555$valuseritempref110144.0

110339.0110433.5110231.5110618.0110515.511075.0210145.5210341.5210436.0210232.5210620.5210515.521074.0310140.0310438.0310526.0310324.5310218.5310616.5310715.5410163.0410455.0410353.5410237.0410633.0410526.041079.5510168.0510459.0510356.5510242.5510634.5510532.0510711.5rmr2使用提示：rmr.options(backend='hadoop')這里backend有兩個值，hadoop,local。hadoop是默認(rèn)值，使用hadoop環(huán)境運行程序。local是一個本地測試的設(shè)置，已經(jīng)不建議再使用。我在開發(fā)時，試過local設(shè)置，運行速度非常快，模擬了hadoop的運行環(huán)境。但是，local模式下的代碼，不能和hadoop模式下完全兼容，變動也比較大，因此不建議大家使用。equijoin(...,outer=c('left'))這里outer包括了4個值，c("”,“l(fā)eft”,“right”,“full”)，非常像數(shù)據(jù)庫中兩個表的join操作

keyval(k,v)mapReduce的操作，需要key和valve保存數(shù)據(jù)。如果直接輸出，或者輸出的未加key，會有一個警告Convertingto.dfsargumenttokeyvalwithaNULLkey。再上一篇文章中，rmr2的例子中就有類似的情況，請大家注意修改代碼。to.dfs(1:10)Warningmessage:Into.dfs(1:10):Convertingto.dfsargumenttokeyvalwithaNULLkey代碼部分：#加載rmr2包library(rmr2)#輸入數(shù)據(jù)文件train<-read.csv(file="small.csv”,header=FALSE)names(train)<-c("user”,"item”,"pref")#使用rmr的hadoop格式，hadoop是默認(rèn)設(shè)置。rmr.options(backend='hadoop')#把數(shù)據(jù)集存入HDFStrain.hdfs=to.dfs(keyval(train$user,train))from.dfs(train.hdfs)from.dfs(train.hdfs)13/04/0714:35:44INFOutil.NativeCodeLoader:Loadedthenative-hadooplibrary13/04/0714:35:44INFOzlib.ZlibFactory:Successfullyloaded&initializednative-zliblibrary13/04/0714:35:44INFOcompress.CodecPool:Gotbrand-newdecompressor$key[1]111222233334444555555$valuseritempref111015.0211023.0311032.5421012.0521022.5621035.0721042.0831012.0

931044.01031054.51131075.01241015.01341033.01441044.51541064.01651014.01751023.01851032.01951044.02051053.52151064.0#STEP1,建立物品的同現(xiàn)矩陣#1)按用戶分組，得到所有物品出現(xiàn)的組合列表。train.mr<-mapreduce(train.hdfs,map=function(k,v){keyval(k,v$item)},reduce=function(k,v){m<-merge(v,v)keyval(m$x,m$y)})from.dfs(train.mr)$key[1]101101101101101101101101101101101101101101101102102102102[20]102102102103103103103103103103103103103103104104104104104[39]104104104104104104104105105105105106106106106107107107107[58]101101101101101101102102102102102102103103103103103103104[77]104104104104104105105105105105105106106106106106106$val[1]101102103101102103104101104105107101103104106101102103101[20]102103104101102103101102103104101103104106101102103104101[39]104105107101103104106101104105107101103104106101104105107[58]101102103104105106101102103104105106101102103104105106101[77]102103104105106101102103104105106101102103104105106#2）對物品組合列表進行計數(shù)，建立物品的同現(xiàn)矩陣

step2.mr<-mapreduce(train.mr,map=function(k,v){d<-data.frame(k,v)d2<-ddply(d,,(k,v),count)key<-d2$kval<-d2keyval(key,val)})from.dfs(step2.mr)$key[1]101101101101101101101102102102102102102103103103103103103[20]104104104104104104104105105105105105105105106106106106106[39]106107107107107$valkvfreq110110152101102331011034410110445101105261011062710110718102101391021023101021033111021042121021051131021061141031014151031023161031034171031043181031051191031062201041014211041022221041033231041044241041052

251041062261041071271051012281051021291051031301051042311051052321051061331051071341061012351061021361061032371061042381061051391061062401071011411071041421071051431071071#2.建立用戶對物品的評分矩陣train2.mr<-mapreduce(train.hdfs,map=function(k,v){#df<-v[which(v$user==3),]df<-vkey<-df$itemval<-data.frame(item=df$item,user=df$user,pref=df$pref)}…y'val)…si,)$key[1]101101101101101102102102103103103103104104104104105105106[20]106107$valitemuserpref110115.0210122.0310132.0410145.0

510154.0610213.0710222.5810253.0910312.51010325.01110343.01210352.01310422.01410434.01510444.51610454.01710534.51810553.51910644.02010654.02110735.0#3.合并同現(xiàn)矩陣和評分矩陣eq.hdfs<-equijoin(left.input=step2.mr,right.input=train2.mr,map.left=function(k,v){廣i"map.right=function(k,v){k'v)outer=c("left")from.dfs(eq.hdfs)$keyNULL$valk.lv.lfreq.litem.ruser.rpref.r1103101410312.52103102310312.53103103410312.54103104310312.55103105110312.56103106210312.57103101410325.0

8103102310325.09103103410325.010103104310325.011103105110325.012103106210325.013103101410343.014103102310343.015103103410343.016103104310343.017103105110343.018103106210343.019103101410352.020103102310352.021103103410352.022103104310352.023103105110352.024103106210352.025101101510115.026101102310115.027101103410115.028101104410115.029101105210115.030101106210115.031101107110115.032101101510122.033101102310122.034101103410122.035101104410122.036101105210122.037101106210122.038101107110122.039101101510132.040101102310132.041101103410132.042101104410132.043101105210132.044101106210132.045101107110132.046101101510145.047101102310145.048101103410145.049101104410145.050101105210145.051101106210145.052101107110145.053101101510154.054101102310154.055101103410154.056101104410154.057101105210154.058101106210154.059101107110154.060105101210534.561105102110534.562105103110534.563105104210534.564105105210534.565105106110534.566105107110534.567105101210553.568105102110553.569105103110553.570105104210553.571105105210553.572105106110553.573105107110553.574106101210644.075106102110644.076106103210644.077106104210644.078106105110644.079106106210644.080106101210654.081106102110654.082106103210654.083106104210654.084106105110654.085106106210654.086104101410422.087104102210422.088104103310422.089104104410422.090104105210422.091104106210422.092104107110422.093104101410434.094104102210434.095104103310434.096104104410434.097104105210434.098104106210434.099104107110434.0100104101410444.5101104102210444.5102104103310444.5103104104410444.5104104105210444.5105104106210444.5106104107110444.5107104101410454.0108104102210454.0109104103310454.0110104104410454.0111104105210454.0112104106210454.0113104107110454.0114102101310213.0115102102310213.0116102103310213.0117102104210213.0118102105110213.0119102106110213.0120102101310222.5121102102310222.5122102103310222.5123102104210222.5124102105110222.5125102106110222.5126102101310253.0127102102310253.0128102103310253.0129102104210253.0130102105110253.0131102106110253.0132107101110735.0133107104110735.0134107105110735.0135107107110735.0#4.計算推薦結(jié)果列表cal.mr<-mapreduce(input=eq.hdfs,

map=function(k,v){val<-vna<-is.na(v$user.r)if(length(which(na))>0)val<-v[-which(is.na(v$user.r)),]keyval(val$k.l,val)},reduce=function(k,v){val<-ddply(v,.(k.l,v.l,user.r),summarize,v=freq.l*pref.r)keyval(val$k.l,val)})from.dfs(cal.mr)$key[1]101101101101101101101101101101101101101101101101101101[19]101101101101101101101101101101101101101101101101101102[37]102102102102102102102102102102102102102102102102102103[55]103103103103103103103103103103103103103103103103103103[73]103103103103103104104104104104104104104104104104104104[91]104104104104104104104104104104104104104104104105105105[109]105105105105105105105105105105105106106106106106106106[127]106106106106106107107107107$valk.lv.luser.rv1101101125.02101101210.03101101310.04101101425.05101101520.06101102115.0710110226.0810110236.09101102415.010101102512.011101103120.01210110328.01310110338.014101103420.015101103516.016101104120.01710110428.01810110438.019101104420.0

20101104516.021101105110.02210110524.02310110534.024101105410.02510110558.026101106110.02710110624.02810110634.029101106410.03010110658.03110110715.03210110722.03310110732.03410110745.03510110754.03610210119.03710210127.53810210159.03910210219.04010210227.54110210259.04210210319.04310210327.54410210359.04510210416.04610210425.04710210456.04810210513.04910210522.55010210553.05110210613.05210210622.55310210653.054103101110.05510310