Mailman 3 January 2018 - scikit-learn

CountVectorizer: Additional Feature Suggestion
by Yacine MAZARI Jan. 31, 2018

Jan. 31, 2018

Hello, I would like to work on adding an additional feature to "sklearn.feature_extraction.text.CountVectorizer". In the current implementation, the definition of term frequency is the number of times a term t occurs in document d. However, another definition that is very commonly used in practice is the term frequency adjusted for document length <https://en.wikipedia.org/wiki/Tf%E2%80%93idf#Term_frequency_2>, i.e: tf = raw counts / document length. I intend to implement this by adding an additional boolean parameter "relative_frequency" to the constructor of CountVectorizer. If the parameter is true, normalize X by document length (along x=1) in "CountVectorizer.fit_transform()". What do you think? If this sounds reasonable an worth it, I will send a PR. Thank you, Yacine.

6 12

DBSCAN Border Points
by AMIR SHANEHSAZZADEH Jan. 31, 2018

Jan. 31, 2018

Hello, I am working with the latest implementation of DBSCAN. I believe that scikit-learn's implementation does not include non-core points in clusters. This results in border points not being included in clusters. Is there any way to remedy this issue so that border points are included in their respective clusters? Do you know what modifications I would need to make the code? Thank you, Amir Shanehsazzadeh

2 1

Fwd: Re: Topic for thesis work on scikit learn
by Gaurav Dhingra Jan. 25, 2018

Jan. 25, 2018

-------- Forwarded Message -------- Subject: Re: [scikit-learn] Topic for thesis work on scikit learn Date: Tue, 23 Jan 2018 10:16:36 -0500 From: Andreas Mueller <t3kcit(a)gmail.com> To: Gaurav Dhingra <gauravdhingra.gxyd(a)gmail.com> Hi Gaurav. Is your mentor experienced in contributing to sklearn? Will they be able to review your code to the scikit-learn standards? Have you worked on any other pull requests so far? Getting anything into scikit-learn without close collaboration with the community is quite tricky. Having a faster K-means implementation based on recent research in the area would be interesting, There's also interest in adding Robust PCA, probabilistic inference trees, and improving the latent dirichlet alloctation code. You can find issues on any of these in the issue tracker, which also has many more feature requests. Andy On 12/31/2017 05:46 AM, Gaurav Dhingra wrote: > > Hi Andreas, > > I think I'll get access to a local mentor from my college, so I think > I rule that issue out, though for technicalities still I would /like/ > to be more dependent on feedback from the scikit-learn community, > since my aim wouldn't be to make something for my own use but rather > something that would be more useful for the scikit-learn community, so > that it eventually gets merged into master. > > I'm currently looking for topic that I can take up, I tried looking > into scikit-learn wiki but it doesn't mention for what I'm looking for > (no topic is mentioned). Do you have some topic in mind that could be > useful for addition to scikit-learn? Even if you could direct me to > appropriate links I would be happy to look into those. > > > On Wednesday 01 November 2017 01:43 AM, Andreas Mueller wrote: >> Hi Gaurav. >> >> Do you have a local mentor? I think having a mentor that can guide >> you during a thesis is very important. >> You could get some feedback from the community for a contribution, >> but that can be slow, >> and is entirely on volunteer basis, so there is no guarantee that >> you'll get the necessary feedback in time >> to finish your thesis. >> >> Mentoring a thesis - in particular without knowing you - is a serious >> commitment, so I'm not sure someone >> from inside the project will want to do this. I saw you already made >> a contribution in >> https://github.com/scikit-learn/scikit-learn/pull/10005 >> but that's a very different scope than doing what I expect would be >> several month of work. > > Though in this regard I've made a few more contributions, here is the > link https://github.com/scikit-learn/scikit-learn/pulls/gxyd, though I > know none of them is a big contribution. If you think I should work on > a big enough PR, can you please suggest me some issue in that regard? > > Thanks. > >> >> >> Best, >> Andy >> >> On 10/31/2017 03:31 PM, Gaurav Dhingra wrote: >>> Hi everyone, >>> >>> I am a final year (5th year) undergraduate Applied Mathematics >>> student in India. I am thinking of doing my final year thesis by >>> doing some work (coding part) on scikit learn, so I was thinking if >>> anyone could tell me if there are available topics (not necessarily >>> names of those topics) that I could work on being an undergraduate >>> student? I would want to expand upon this in December when my exams >>> will be over. But in the mean time would want to take a step in that >>> direction by just knowing if there will be available topics that I >>> could work on. >>> >>> It could be the case that available topics are not so easy for an >>> undergraduate, still in that case I would like to do some research >>> on the topics first. >>> >> >> _______________________________________________ >> scikit-learn mailing list >> scikit-learn(a)python.org >> https://mail.python.org/mailman/listinfo/scikit-learn > > -- > Gaurav Dhingra > (sent from Thunderbird email client)

2 2

GSoC 2018
by Aman Pratik Jan. 17, 2018

Jan. 17, 2018

Hello Admins, I wished to know if scikit-learn would participate in GSoC this year (2018). Thanks, Aman Pratik

1 0

How to train an image classifier on directories
by Abdul Abdul Jan. 15, 2018

Jan. 15, 2018

Hello, I'm trying to train an image classifier, but a bit confused on how to label my data. The issue here is that for each class I have subdirectories, each of which contains two images. So, it is not I have classes, and in each class I simply have the images that come under that class (i.e. cats vs. dogs). I will show here some attempts for grouping the data together, but not yet able to figure how to assign the label, and pass the pairs of images along with the label to the image classifier. So, that's how I simply read the two images: im1 = cv2.imread('img1.jpg') im1 = img_to_array(im1) im2 = cv2.imread('img2.jpg') im2 = img_to_array(im2) I then *pair* the images as follows: pair = (im1,im2) For labeling, this is what I did: label = root.split(os.path.sep)[-2] label = 1 if label == 'cat' else 0 How can I group the above pairs of images (im1,im2) and attach the label to them? Especially that I want to pass them to the following scikit-learn function: (trainX, testX, trainY, testY) = train_test_split(data, labels, test_size=0.25, random_state=42) Thanks.

2 1

MPLclassifier
by andreas heiner Jan. 14, 2018

Jan. 14, 2018

Hi, I try to apply the MPLclassifier to a subset (100 data points, 2 classes) of the 20newsgroup dataset. I created (ok, copied) the following pipeline model_MLP = Pipeline([('vect', CountVectorizer()), ('tfidf', TfidfTransformer()), ('model_MLP', MLPClassifier(solver='lbfgs', alpha=1e-5, hidden_layer_sizes=(5, 2), random_state=1) ) ]) model_MLP.fit(twenty_train.data, twenty_train.target) predicted_MLP = model_MLP.predict(twenty_test.data) print(metrics.classification_report(twenty_test.target, predicted_MLP, target_names=twenty_test.target_names)) The numbers I get are hopeless, precision recall f1-score support alt.atheism 0.00 0.00 0.00 34 sci.electronics 0.66 1.00 0.80 66 The only reason I can think of is that the dictionaries of the training and the test set are not the same (testset: 5204 words, training set: 5402 words). That should not be a problem (if I understand Bayes correctly), but it certainly gives rubbish (see the numbers). The same setup with the SVD routine works great, all values are around .95 thanks, Andreas

2 1

Re: [scikit-learn] sklearn.model_selection.GridSearchCV - unable to use n_jobs>1 on MacOS Sierra python 2.7
by Sumeet Sandhu Jan. 10, 2018

Jan. 10, 2018

There are two cases : n_jobs > 1 works when data is smaller - when the training docs numpy array is 15MB. It does not work when training matrix is 100MB. My Mac has 16GB RAM. In the second case, the jobs die out pretty quickly, in seconds, and the main python process seems to die out (min CPU usage). There is a popup message saying 'python processes appear to have died'. This is when i run python on bash command line. When I run in python GUI IDLE, a message pops up 'your program is still running, sure you want to close window'. What are these jobs anyway? Are they various parameter combinations in param_grid, or lower level jobs out of compiler etc? Does each job replicate the training data in RAM? regards On Sun, Jan 7, 2018 at 11:35 AM, Sumeet Sandhu <sumeet.k.sandhu(a)gmail.com> wrote: > Hi, > > I was able to run this with n_jobs=-1, and the activity monitor does show > all 8 CPUs engaged, but the jobs start to die out one by one. I tried with > n_jobs=2, same story. > The only option that works is n_jobs=1. > I played around with 'pre_dispatch' a bit - unclear what that does. > > GRID = GridSearchCV(LogisticRegression(), param_grid, scoring=None, > fit_params=None, n_jobs=1, iid=True, refit=True, cv=10, verbose=0, > error_score=0, return_train_score=False) > GRID.fit(trainDocumentV,trainLabelV) > > > How can I sustain at least 3-4 parallel jobs? > > thanks, > Sumeet >

1 1

Any plans on generalizing Pipeline and transformers?
by Manuel Castejón Limas Jan. 9, 2018

Jan. 9, 2018

Dear all, Kudos to scikit-learn! Having said that, Pipeline is killing me not being able to transform anything other than X. My current study case would need: - Transformers being able to handle both X and y, e.g. clustering X and y concatenated - Pipeline being able to change other params, e.g. sample_weight Currently, I'm augmenting X through every step with the extra information which seems to work ok for my_pipe.fit_transform(X_train,y_train) but breaks on my_pipe.transform(X_test) for the lack of the y parameter. Ok, I can inherit and modify a descendant from Pipeline class to allow the y parameter which is not ideal but I guess it is an option. The gritty part comes when having to adapt every regressor at the end of the ladder in order to split the extra information from the raw data in X and not being able to generate more than one subproduct from each preprocessing step My current research involves clustering the data and using that classification along with X in order to predict outliers which generates sample_weight info and I would love to use that on the final regressor. Currently there seems not to be another option than pasting that info on X. All in all, I'm stuck with this API limitation and I would love to learn some tricks from you if you could enlighten me. Thanks in advance! Manuel Castejón-Limas

4 6

sklearn.model_selection.GridSearchCV - unable to use n_jobs>1 on MacOS Sierra python 2.7
by Sumeet Sandhu Jan. 8, 2018

Jan. 8, 2018

Hi, I was able to run this with n_jobs=-1, and the activity monitor does show all 8 CPUs engaged, but the jobs start to die out one by one. I tried with n_jobs=2, same story. The only option that works is n_jobs=1. I played around with 'pre_dispatch' a bit - unclear what that does. GRID = GridSearchCV(LogisticRegression(), param_grid, scoring=None, fit_params=None, n_jobs=1, iid=True, refit=True, cv=10, verbose=0, error_score=0, return_train_score=False) GRID.fit(trainDocumentV,trainLabelV) How can I sustain at least 3-4 parallel jobs? thanks, Sumeet

3 2

clustering on big dataset
by Shiheng Duan Jan. 5, 2018

Jan. 5, 2018

Hi all, I wonder if there is any method to do exact clustering (hierarchy cluster) on a huge dataset where it is impossible to use distance matrix. I am considering KD-tree but every time it needs to rebuild it, consuming lots time. Any ideas?

3 5