Code Stop #68
Description
Below is my log of running the code. The code stops at the last line so long time, there's no error. I don't know why:
...
Initializing [clip] visual encoder from ~/.cache/clip/ViT-L-14.pt [cuda]...
Visual encoder initialized.
Initializing language decoder from /home/project/Mllm/LAMM/model_zoo/vicuna_ckpt/vicuna-13b-delta-v0 ...
Visual encoder initialized.
Initializing language decoder from /home/project/Mllm/LAMM/model_zoo/vicuna_ckpt/vicuna-13b-delta-v0 ...
Visual encoder initialized.
Initializing language decoder from /home/project/Mllm/LAMM/model_zoo/vicuna_ckpt/vicuna-13b-delta-v0 ...
Visual encoder initialized.
Initializing language decoder from /home/project/Mllm/LAMM/model_zoo/vicuna_ckpt/vicuna-13b-delta-v0 ...
Loading checkpoint shards: 100%|██████████| 3/3 [00:17<00:00, 5.86s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:19<00:00, 6.53s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:20<00:00, 6.96s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:20<00:00, 6.71s/it]
trainable params: 52428800 || all params: 13068303360 || trainable%: 0.4011905643426998
You are using the legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This means that tokens that come after special tokens will not be properly handled. We recommend you to read the related pull request available at huggingface/transformers#24565
Language decoder initialized.
LLaMa projection layer initialized.
DeepSpeedAgent
trainable params: 52428800 || all params: 13068303360 || trainable%: 0.4011905643426998
You are using the legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This means that tokens that come after special tokens will not be properly handled. We recommend you to read the related pull request available at huggingface/transformers#24565
Language decoder initialized.
LLaMa projection layer initialized.
DeepSpeedAgent
trainable params: 52428800 || all params: 13068303360 || trainable%: 0.4011905643426998
You are using the legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This means that tokens that come after special tokens will not be properly handled. We recommend you to read the related pull request available at huggingface/transformers#24565
Language decoder initialized.
LLaMa projection layer initialized.
DeepSpeedAgent
trainable params: 52428800 || all params: 13068303360 || trainable%: 0.4011905643426998
You are using the legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This means that tokens that come after special tokens will not be properly handled. We recommend you to read the related pull request available at huggingface/transformers#24565
Language decoder initialized.
LLaMa projection layer initialized.
DeepSpeedAgent
[2024-01-24 16:43:41,259] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.9.3, git-hash=4e80e29, git-branch=HEAD
[2024-01-24 16:43:41,259] [INFO] [comm.py:619:init_distributed] Distributed backend already initialized
[2024-01-24 16:44:17,242] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
Using /home/.cache/torch_extensions/py310_cu118 as PyTorch extensions root...
Using /home/.cache/torch_extensions/py310_cu118 as PyTorch extensions root...
Using /home/.cache/torch_extensions/py310_cu118 as PyTorch extensions root...
Using /home/.cache/torch_extensions/py310_cu118 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/.cache/torch_extensions/py310_cu118/cpu_adam/build.ninja...
Building extension module cpu_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/3] /usr/bin/nvcc -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -I/data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/deepspeed/ops/csrc/includes -I/usr/include -isystem /data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include -isystem /data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include/TH -isystem /data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include/THC -isystem /data2/miniconda3/envs/lamm3/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_90,code=compute_90 -gencode=arch=compute_90,code=sm_90 --compiler-options '-fPIC' -O3 --use_fast_math -std=c++17 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_HALF2_OPERATORS__ -gencode=arch=compute_90,code=sm_90 -gencode=arch=compute_90,code=compute_90 -DBF16_AVAILABLE -c /data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/deepspeed/ops/csrc/common/custom_cuda_kernel.cu -o custom_cuda_kernel.cuda.o
[2/3] c++ -MMD -MF cpu_adam.o.d -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -I/data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/deepspeed/ops/csrc/includes -I/usr/include -isystem /data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include -isystem /data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include/TH -isystem /data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/include/THC -isystem /data2/miniconda3/envs/lamm3/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O3 -std=c++17 -g -Wno-reorder -L/usr/lib64 -lcudart -lcublas -g -march=native -fopenmp -D__AVX512 -D__ENABLE_CUDA_ -DBF16_AVAILABLE -c /data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/deepspeed/ops/csrc/adam/cpu_adam.cpp -o cpu_adam.o
[3/3] c++ cpu_adam.o custom_cuda_kernel.cuda.o -shared -lcurand -L/data2/miniconda3/envs/lamm3/lib/python3.10/site-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda -ltorch -ltorch_python -L/usr/lib64 -lcudart -o cpu_adam.so
Loading extension module cpu_adam...
Time to load cpu_adam op: 23.828491926193237 seconds
Loading extension module cpu_adam...
Time to load cpu_adam op: 23.78138303756714 seconds
Loading extension module cpu_adam...
Time to load cpu_adam op: 23.877806186676025 seconds
Loading extension module cpu_adam...
Time to load cpu_adam op: 23.85369896888733 seconds