Quantcast
Channel: Active questions tagged r - Stack Overflow
Viewing all articles
Browse latest Browse all 201945

R function to import text data in predefined Document Term Matrix

$
0
0

I have imported some text data regarding description of some products. That text data was transformed to corpus, made some data preprocessing and finally transfer it to Document Term Matrix (as data frame) to create a CART model. Here is the code:

#Data load
o.data = read_excel("ML data.xlsx")

# Preparation
o.data$label = as.character(o.data$label)
corpus = Corpus(VectorSource(o.data$label))
corpus = tm_map(corpus, tolower)
corpus = tm_map(corpus, removePunctuation)
corpus_matrix = DocumentTermMatrix(corpus)
sparse = removeSparseTerms(corpus_matrix, 0.995)
m.data = as.data.frame(as.matrix(sparse))

After that I have loaded ID and independent variable to the DTM data set, split the data to training and testing set, build a model and made good predictions on the testing set with this code:

colnames(m.data) = make.names(colnames(m.data))
# Add independent variable (k101) & id variable
m.data$k101 = o.data$k101
m.data$id = o.data$id_new
split = sample.split(m.data, SplitRatio = 0.7)
m.data_train = subset(m.data, split==TRUE)
m.data_test = subset(m.data, split==FALSE)
CART_model_k101 = rpart(k101~., data=m.data_train, method="class")
#...and so on to the model evaluation etc.

Now the problem is when I try to import new text data, convert it to corpus and Document Term Matrix (as data frame), I get different DTM matrix and therefore can't use my previously built model to classify new text data. Here is the code:

t.data = read_excel("new_text_data.xlsx")
corpus_test = Corpus(VectorSource(t.data$label_test))
corpus_test = tm_map(corpus_test, tolower)
corpus_test = tm_map(corpus_test, removePunctuation)
corpus_matrix_test = DocumentTermMatrix(corpus_test)
sparse_test = removeSparseTerms(corpus_matrix_test, 0.995)
corpus.test.data = as.data.frame(as.matrix(sparse_test))
corpus.test.data$k101_pvalue = predict(CART_model_k101, newdata=corpus.test.data)[,2]

After I attempt to made prediction on new text data using the model which was build using the DTM from previous text data I get an error where object['column name'] is not found - it searches for the exact same DTM structure (column names) in the new DTM on testing data.

My problem is to find a way where I can transfer corpus from the testing data to the exact DTM structure which I have from the training data.

I apologize for a lot of text and explanation and hope that there is a solution to this.


Viewing all articles
Browse latest Browse all 201945

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>