WIP - Add dimension reduction performance tests by jamesryan094 · Pull Request #13 · microscopium/microscopium-scripts

jamesryan094 · 2018-10-05T00:07:28Z

Add work in progress script for analysing performance of PCA, tSNE and UMAP on BBBC021 dataset

…d UMAP on BBBC021 dataset

jni

Hello!

I've made a few comments, mostly nitpicky stuff about syntax/style, with just one suggestion for improved efficiency/effectiveness.

Let's get this merged before anything, but longer term, we should:

(a) allow the app to use any pair of columns to plot against each other in the scatter plot
(b) define a standard format, consisting of just a csv (or other, more efficient columnar format, for example a database or a parquet file) with at least the fields "index,info,url" (and nothing else), and the corresponding images (in any arbitrary directory structure, as long as "url" corresponds to the right image).
(c) provide a general function/command-line utility to go from (b) to a csv as "index,info,url,feat0,feat1,feat2..."
(d) provide a general function/utility to go from (c) to "index,info,url,pca0,pca1,tsne0,tsne1,umap0,umap1"

With these defined data points, it should be a lot easier to ingest new datasets: we just need to figure out how to get a new dataset to point (b), and then everything downstream of that is standard.

jni · 2018-10-05T05:59:19Z

+    """
+    Construct filenames for the montaged images
+
+    :param filenames: names of the illumed images with separate quadrants and channels


Can you switch your default docstring formatting to NumPy style? I think I saw you using PyCharm? It has NumPy docstring style as an option.

jni · 2018-10-05T06:10:47Z

+    :param filenames: names of the illumed images with separate quadrants and channels
+    :return names_montage: names for each file which will result from montaging
+    """
+    filename_reg = r'(.*Week._.*_)(...)(_s._w..*)(_illum)(\.tif)$'


Mad regex skillz! =D https://twitter.com/iamdevloper/status/580326082269843456

jni · 2018-10-05T06:22:30Z

+def get_valid_file_names(filepath):
+    """
+    Get full filenames relative to top level directory for each file in the BBBC trial, and
+    construct filenames with paths for saving output


Please read and conform to PEP 257. Specifically, first docstring line should be inline with the triple quotes and fit in 80c. You can expand more after a blank line.

Link: https://www.python.org/dev/peps/pep-0257/

jni · 2018-10-05T06:23:32Z

+        for filename in os.listdir(current_subdir):
+            match = re.search(filename_reg, filename)
+            if match:
+               new_subdir.append(os.path.join(root, match.group(1) + match.group(2) + match.group(3) + match.group(4)))


The indentation here and below is funky. Always indent by four spaces! And limit text to 80c width. I think PyCharm (and most Python editors) have settings to warn you about this sort of stuff automatically.

jni · 2018-10-05T06:25:12Z

+    illumed_ims = map(io.imread, filenames)
+    montaged_ims = montage_stream(illumed_ims, montage_order=[[0, 1], [2, 3]], channel_order=[2, 1, 0])
+    for (image, name) in zip(montaged_ims, names_out):
+        io.imsave(name, image)


A very nice function! =)

jni · 2018-10-05T06:29:04Z

+            flag = False
+        image_features = pd.DataFrame(image_features).transpose()
+        all_image_features = all_image_features.append(image_features, ignore_index=True)
+        print("Image processed: {} out of {}".format(i, len(filenames)))


Didn't I mention tqdm to you guys? =)

Link: https://tqdm.github.io/

jni · 2018-10-05T06:33:05Z

+        if flag:
+            all_image_features = all_image_features.append(pd.DataFrame(feature_names).transpose())
+            flag = False
+        image_features = pd.DataFrame(image_features).transpose()


So I think this is probably slowing you down: appending to a dataframe is slow. You should keep the header from the first image, and then append feature arrays to a list. Then, only at the very end, concatenate all the feature arrays into a big one (using np.stack(arrays_list)), and make a pandas dataframe using the big array and the column names.

An even better alternative is to never make the dataframe in the first place: open a file, write the comma-separated header, then write the values of each row to the file. That way, if the project crashes, you don't have to start from scratch again!

I can also attest that appending to dataframes is very slow. I didn't pick up on this as a potential issue when I looked at the code before, sorry. I recently wrote a script that ran six times faster when I switched appending to a dataframe to appending to lists instead.

Please use the "with open" method if you are going to write to file, shown here as best practise in the little book of python anitpatterns: https://docs.quantifiedcode.com/python-anti-patterns/maintainability/not_using_with_to_open_files.html?highlight=open

jni · 2018-10-05T06:33:57Z

+    """
+    all_image_features = pd.read_csv(features_filename)
+
+    return umap.UMAP().fit_transform(all_image_features.iloc[1:, 2:])


Why are you discarding the first row?

Because currently the dataframe column headings are being written as though they were a row of data, instead of column names. That's ok, it's been noted elsewhere.

jni · 2018-10-05T06:34:32Z

+    """
+    Generate a CSV of columns
+        index,info,url,x,y
+    to work with Bokeh app.


GenevieveBuckley · 2018-10-05T07:34:36Z

+    filenames_col = pd.DataFrame(filenames_col)
+
+    all_image_features = pd.DataFrame()
+    # set up flag to only add header row once


Like I mentioned earlier (and I know you haven't had time to do anything yet), this logic with the flag is clever but unnecessary - pandas has a keyword argument "columns" for defining the header row.

df = pd.DataFrame(columns=['col1', 'col2', 'col3']) . # returns an empty dataframe with column headings 'col1', 'col2', 'col3' df = pd.DataFrame(data, columns=['col1', 'col2', 'col3']) # returns a dataframe with data and column headings

https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html

GenevieveBuckley · 2018-10-05T07:37:53Z

+OUTPUT_FILE_PATH = "/data/bbbc_out/"
+
+FEATURES_FILE = "./all_features.csv"
+DATA_FILE = OUTPUT_FILE_PATH + "Data.csv"


Should this be DATA_PCA and "Data_PCA.csv" for consistency?

GenevieveBuckley · 2018-10-05T07:39:14Z

+   #      print("Directory {} out of {} processed".format(i, len(names_illum)))
+
+
+     ## run features on images


More funky whitespace (should be 4 spaces, not 5 here)

GenevieveBuckley · 2018-10-05T07:39:21Z

+    ims = map(io.imread, all_names_montage)
+    output_features(ims, all_names_montage, FEATURES_FILE)
+
+     ## get x y coordinates


More funky whitespace (should be 4 spaces, not 5 here)

GenevieveBuckley · 2018-10-05T07:39:27Z

+     ## get x y coordinates
+    coords_pca = pca_transform(FEATURES_FILE)
+    print("Coords PCA retrieved")
+     ## generate CSV of coordinates


More funky whitespace (should be 4 spaces, not 5 here)

GenevieveBuckley · 2018-10-08T00:29:39Z

+    coord_csv.columns = ["index", "info", "url", "x", "y"]
+    coord_csv.to_csv(filename)
+
+main()


Suggest using:

if __name__ == '__main__': main()

to allow re-use of functions by importing them into other scripts. Might be worth the extra flexibility.

…ix indenting.

Add work in progress script for analysing performance of PCA, tSNE an…

67f1225

…d UMAP on BBBC021 dataset

jni reviewed Oct 5, 2018

View reviewed changes

GenevieveBuckley reviewed Oct 5, 2018

View reviewed changes

GenevieveBuckley reviewed Oct 8, 2018

View reviewed changes

Add numpy docstrings. Normalise features. Add compound info to csv. F…

054b1d9

…ix indenting.

		# print("Directory {} out of {} processed".format(i, len(names_illum)))


		## run features on images

Uh oh!

Conversation

jamesryan094 commented Oct 5, 2018

Uh oh!

jni left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants