TypeError: Object Of Type 'numpy.int64' Has No Len()

I am making a DataLoader from DataSet in PyTorch. Start from loading the DataFrame with all dtype as an np.float64 result = pd.read_csv('dummy.csv', header=0, dtype=DTYPE_CLEANED_

Solution 1:

I think the issue is that after using random_split, index is now a torch.Tensor rather than an int. I found that adding a quick type check to __getitem__ and then using .item() on the tensor works for me:

def __getitem__(self, index):

    if type(index) == torch.Tensor:
        index = index.item()

    x = torch.tensor(self.x_data.iloc[index].values, dtype=torch.float)
    y = torch.tensor(self.y_data.iloc[index], dtype=torch.float)
    return (x, y)

Source: https://discuss.pytorch.org/t/issues-with-torch-utils-data-random-split/22298/8

Solution 2:

Reference:
https://github.com/pytorch/pytorch/issues/9211

Just add .tolist() to indices line.

def random_split(dataset, lengths):
    """
    Randomly split a dataset into non-overlapping new datasets of given lengths.
    Arguments:
        dataset (Dataset): Dataset to be split
        lengths (sequence): lengths of splits to be produced
    """
    if sum(lengths) != len(dataset):
        raise ValueError("Sum of input lengths does not equal the length of the input dataset!")

    indices = randperm(sum(lengths)).tolist()
    return [Subset(dataset, indices[offset - length:offset]) for offset, length in zip(_accumulate(lengths), lengths)]

Solution 3:

Why not simply to try:

self.len = len(self.x_data)

len works fine with pandas DataFrame w/o conversion to array or tensor.

Solution 4:

I solved the issue by upgrading my version of PyTorch to version 1.3.

https://pytorch.org/get-started/locally/

Solution 5:

I have total 2298 images. So if I do the following way

[int(len(data)*0.8),int(len(data)*0.2)]

it throw the error mentioned in question. As

[int(len(data)*0.8)+int(len(data)*0.2)]=2297

So what I do is floor and ceil functions

[int(np.floor(len(data)*0.8)),int(np.ceil(len(data)*0.2))])

and it resulted in 2298 and error is gone