Nan error on cSCC dataseet

[Holly-Wang]

Hi. Thank you for your nice work. I'm having problems with nan in running HGGEP_train.py. Specifically, I run python HGGEP_train.py --data "cscc". An error occured:
"Traceback (most recent call last): | 0/1 [00:00<?, ?it/s]
File "/d/wangx/HGGEP/HGGEP_train.py", line 89, in
trainer.fit(model, train_loader, test_loader)
File "/d/wangx/miniconda3/envs/clip_prediction/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 608, in fit
call._call_and_handle_interrupt(
File "/d/wangx/miniconda3/envs/clip_prediction/lib/python3.9/site-packages/pytorch_lightning/trainer/call.py", line 38, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/d/wangx/miniconda3/envs/clip_prediction/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 650, in _fit_impl
self._run(model, ckpt_path=self.ckpt_path)
File "/d/wangx/miniconda3/envs/clip_prediction/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1103, in _run
results = self._run_stage()
File "/d/wangx/miniconda3/envs/clip_prediction/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1182, in _run_stage
self._run_train()
File "/d/wangx/miniconda3/envs/clip_prediction/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1205, in _run_train
self.fit_loop.run()
File "/d/wangx/miniconda3/envs/clip_prediction/lib/python3.9/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
self.advance(*args, **kwargs)
File "/d/wangx/miniconda3/envs/clip_prediction/lib/python3.9/site-packages/pytorch_lightning/loops/fit_loop.py", line 267, in advance
self._outputs = self.epoch_loop.run(self._data_fetcher)
File "/d/wangx/miniconda3/envs/clip_prediction/lib/python3.9/site-packages/pytorch_lightning/loops/loop.py", line 200, in run
self.on_advance_end()
File "/d/wangx/miniconda3/envs/clip_prediction/lib/python3.9/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 250, in on_advance_end
self._run_validation()
File "/d/wangx/miniconda3/envs/clip_prediction/lib/python3.9/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 308, in _run_validation
self.val_loop.run()
File "/d/wangx/miniconda3/envs/clip_prediction/lib/python3.9/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
self.advance(*args, **kwargs)
File "/d/wangx/miniconda3/envs/clip_prediction/lib/python3.9/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 152, in advance
dl_outputs = self.epoch_loop.run(self._data_fetcher, dl_max_batches, kwargs)
File "/d/wangx/miniconda3/envs/clip_prediction/lib/python3.9/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
self.advance(*args, **kwargs)
File "/d/wangx/miniconda3/envs/clip_prediction/lib/python3.9/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 137, in advance
output = self._evaluation_step(**kwargs)
File "/d/wangx/miniconda3/envs/clip_prediction/lib/python3.9/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 234, in _evaluation_step
output = self.trainer._call_strategy_hook(hook_name, *kwargs.values())
File "/d/wangx/miniconda3/envs/clip_prediction/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1485, in _call_strategy_hook
output = fn(*args, **kwargs)
File "/d/wangx/miniconda3/envs/clip_prediction/lib/python3.9/site-packages/pytorch_lightning/strategies/strategy.py", line 390, in validation_step
return self.model.validation_step(*args, **kwargs)
File "/d/wangx/HGGEP/HGGEP.py", line 322, in validation_step
r.append(pearsonr(pred[g], exp[g])[0])
File "/d/wangx/miniconda3/envs/clip_prediction/lib/python3.9/site-packages/scipy/stats/_stats_py.py", line 4793, in pearsonr
normxm = linalg.norm(xm)
File "/d/wangx/miniconda3/envs/clip_prediction/lib/python3.9/site-packages/scipy/linalg/_misc.py", line 146, in norm
a = np.asarray_chkfinite(a)
File "/d/wangx/miniconda3/envs/clip_prediction/lib/python3.9/site-packages/numpy/lib/function_base.py", line 628, in asarray_chkfinite
raise ValueError(
ValueError: array must not contain infs or NaNs". I'd be appreciate if you could help me solve the error.

[Boom5426]

Hello, thank you for your attention.

I have observed that the issue occurs on this line of code: File "/d/wangx/HGGEP/HGGEP.py", line 322, in validation_step: r.append(pearsonr(pred[g], exp[g])[0]). Based on my experience, this is due to the poor performance of the model in the early stages of training, which causes infs/NaN values to appear in pred, thereby leading to the failure of the calculation. You can avoid this by directly commenting out these lines of code for now.

If there are still issues, please contact me or update this issue.

[Holly-Wang]

Thank you for your detailed answer! But I followed your method, saved the checkpoints after training 400 epochs and then called test function. The same error occured. I wonder if it's because I'm using a different version of the packages in my conda environment. I'd so appreciate it if you could upload the yaml file or the full detailed requirements.txt of the specific virtual environment used to train the model.

The error is:
File "/d/wangx/HGGEP/HGGEP_train_normaltrain_skin.py", line 104, in
R=get_R(pred,gt)[0]
File "/d/wangx/HGGEP/predict.py", line 109, in get_R
r,pv=func(adata1[:,g],adata2[:,g])
File "/d/wangx/miniconda3/envs/clip_prediction/lib/python3.9/site-packages/scipy/stats/_stats_py.py", line 4793, in pearsonr
normxm = linalg.norm(xm)
File "/d/wangx/miniconda3/envs/clip_prediction/lib/python3.9/site-packages/scipy/linalg/_misc.py", line 146, in norm
a = np.asarray_chkfinite(a)
File "/d/wangx/miniconda3/envs/clip_prediction/lib/python3.9/site-packages/numpy/lib/function_base.py", line 628, in asarray_chkfinite
raise ValueError(
ValueError: array must not contain infs or NaNs

[Boom5426]

As I have changed my workplace, the original environmental data is no longer available. I noticed that the core issue still lies in the presence of infs or NaNs in the prediction results. Therefore, it is possible to manually clean up these anomalies. You can refer to the code in HGGEP/predict.py, specifically the line # adata1, adata2 = replace_nans_infs_numpy(adata1, adata2).

[blueilsky]

Hi, I’m also trying to reproduce the results on this dataset and have encountered the same issue where the results come out as NaN. I haven’t been able to resolve it. Have you managed to solve it? If so, could you kindly share how you fixed it? I would really appreciate your help.