Kmeans hangs sometimes

[vikramc1]

When using the Kmeans function, there are times when the program just hangs. Cannot reproduce deterministically because it only happens sometimes. but the following code hangs sometimes and requires a restart of the kernel (running in jupyter notebook):

def get_cluster_labels(Z,k):
    cluster_labels = []
    for i in range(Z.shape[0]):
        data = np.squeeze(Z[i,:,:]).reshape(Z.shape[1],Z.shape[2])
        kmeans = Kmeans(data,k)
        cluster_labels.append(kmeans.get_clusters())
    return cluster_labels

[disa-mhembere]

I'll look into this.

What's the shape of Z?
What is your OS?
What are the hardware specs (#cores, memory)?
How many thread of execution are you running?

[vikramc1]

1. Z is of shape (m,150,d) where m can be between 0 and 10000 and d can be between 1 and 4.
2. OS is Ubuntu 18.04.2 LTS
3. machine has 48 cores and 250 GB memory
4. using the default (which i think is num_threads = 48)

[disa-mhembere]

I've synchronized the code from the most up to date C++ commit. Unfortunately, the API has had to change to one similar to sklearn. Try to upgrade then change your code to:

:
_kmeans = Kmeans(k, nthread=48) # You must explicitly set the number of threads
kmeans = _kmeans.fit(data) # You will now receive back a `cluster_t` object
:

Let me know if this resolves your issue as soon as possible.

[vikramc1]

Trying to upgrade my installation of knor using python -m pip install --upgrade knor having already installed setuptools,numpy, and pybind11 and got the following error:

Collecting knor
  Using cached https://files.pythonhosted.org/packages/cb/e7/53cf9d75a7dfc1a81df903f64c4a4f0e02ab12c373c62fa14dd9279178a8/knor-0.0.5.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-g7ud81c8/knor/setup.py", line 11, in <module>
        from setuptools.command.build_clib import build_clib
    ModuleNotFoundError: No module named 'setuptools.command.build_clib'

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-g7ud81c8/knor/

[disa-mhembere]

Not sure why I'm not getting these notifications when you post, but did you already apt install python-setuptools?

[vikramc1]

yup, I already installed setuptools but there are a few versions on my system. Looks like im using setuptools==28.8.0

[disa-mhembere]

Just for documentation this is was an old version of setuptools issue resolved by using the pip managed version instead of apt:

sudo apt remove python-setuptools python3-setuptool
pip install -U setuptools
pip3 install -U setuptools

[vikramc1]

After updating and testing the new version of code, it still hangs sometimes