python tools/predict.py --stage "float" --config configs/bev/detr3d_efficientnetb3_nuscenes.py `aidisdk` dependency is not available. /home/qinyonghang/miniconda3/envs/j6/lib/python3.10/site-packages/torchvision/transforms/functional_tensor.py:5: UserWarning: The torchvision.transforms.functional_tensor module is deprecated in 0.15 and will be **removed in 0.17**. Please don't rely on it. You probably just need to use APIs in torchvision.transforms.functional or in torchvision.transforms.v2.functional. warnings.warn( `aidisdk` dependency is not available. INFO:hat.engine.ddp_trainer:Launch with rank: 0 world_size: None hostname: qkc dist_url: tcp://localhost:11300 num_devices: 1 num_processes: 1 /home/qinyonghang/miniconda3/envs/j6/lib/python3.10/site-packages/torchvision/transforms/functional_tensor.py:5: UserWarning: The torchvision.transforms.functional_tensor module is deprecated in 0.15 and will be **removed in 0.17**. Please don't rely on it. You probably just need to use APIs in torchvision.transforms.functional or in torchvision.transforms.v2.functional. warnings.warn( 2024-06-24 14:35:28,330 INFO [logger.py:199] Node[0] ==================================================BEGIN FLOAT PREDICT================================================== 2024-06-24 14:35:30,894 WARNING: wrap usage has been changed, please pass necessary args 2024-06-24 14:35:31,737 INFO [base_toolkit.py:293] Node[0] Could not load traitsui.toolkits plugin 'qt' from 'traitsui.qt' 2024-06-24 14:35:31,741 INFO [base_toolkit.py:293] Node[0] Could not load traitsui.toolkits plugin 'wx' from 'traitsui.wx' 2024-06-24 14:35:31,743 INFO [base_toolkit.py:293] Node[0] Could not load traitsui.toolkits plugin 'qt4' from 'traitsui.qt' ******************************************************************************** WARNING: Imported VTK version (9.3) does not match the one used to build the TVTK classes (9.2). This may cause problems. Please rebuild TVTK. ******************************************************************************** 2024-06-24 14:35:33,775 INFO [logger.py:199] Node[0] building bifpn cell 0 2024-06-24 14:35:33,776 INFO [logger.py:199] Node[0] fnode 0 : {'inputs_offsets': [3, 4], 'sampling': ['keep', 'up'], 'upsample_type': ['function', 'function']} 2024-06-24 14:35:33,777 INFO [logger.py:199] Node[0] fnode 1 : {'inputs_offsets': [2, 5], 'sampling': ['keep', 'up'], 'upsample_type': ['function', 'function']} 2024-06-24 14:35:33,779 INFO [logger.py:199] Node[0] fnode 2 : {'inputs_offsets': [1, 6], 'sampling': ['keep', 'up'], 'upsample_type': ['function', 'function']} 2024-06-24 14:35:33,780 INFO [logger.py:199] Node[0] fnode 3 : {'inputs_offsets': [0, 7], 'sampling': ['keep', 'up'], 'upsample_type': ['function', 'function']} 2024-06-24 14:35:33,782 INFO [logger.py:199] Node[0] fnode 4 : {'inputs_offsets': [1, 7, 8], 'sampling': ['keep', 'keep', 'down'], 'upsample_type': ['function', 'function', 'function']} 2024-06-24 14:35:33,784 INFO [logger.py:199] Node[0] fnode 5 : {'inputs_offsets': [2, 6, 9], 'sampling': ['keep', 'keep', 'down'], 'upsample_type': ['function', 'function', 'function']} 2024-06-24 14:35:33,786 INFO [logger.py:199] Node[0] fnode 6 : {'inputs_offsets': [3, 5, 10], 'sampling': ['keep', 'keep', 'down'], 'upsample_type': ['function', 'function', 'function']} 2024-06-24 14:35:33,787 INFO [logger.py:199] Node[0] fnode 7 : {'inputs_offsets': [4, 11], 'sampling': ['keep', 'down'], 'upsample_type': ['function', 'function']} 2024-06-24 14:35:33,788 INFO [logger.py:199] Node[0] building bifpn cell 1 2024-06-24 14:35:33,788 INFO [logger.py:199] Node[0] fnode 0 : {'inputs_offsets': [3, 4], 'sampling': ['keep', 'up'], 'upsample_type': ['function', 'function']} 2024-06-24 14:35:33,789 INFO [logger.py:199] Node[0] fnode 1 : {'inputs_offsets': [2, 5], 'sampling': ['keep', 'up'], 'upsample_type': ['function', 'function']} 2024-06-24 14:35:33,791 INFO [logger.py:199] Node[0] fnode 2 : {'inputs_offsets': [1, 6], 'sampling': ['keep', 'up'], 'upsample_type': ['function', 'function']} 2024-06-24 14:35:33,792 INFO [logger.py:199] Node[0] fnode 3 : {'inputs_offsets': [0, 7], 'sampling': ['keep', 'up'], 'upsample_type': ['function', 'function']} 2024-06-24 14:35:33,793 INFO [logger.py:199] Node[0] fnode 4 : {'inputs_offsets': [1, 7, 8], 'sampling': ['keep', 'keep', 'down'], 'upsample_type': ['function', 'function', 'function']} 2024-06-24 14:35:33,794 INFO [logger.py:199] Node[0] fnode 5 : {'inputs_offsets': [2, 6, 9], 'sampling': ['keep', 'keep', 'down'], 'upsample_type': ['function', 'function', 'function']} 2024-06-24 14:35:33,795 INFO [logger.py:199] Node[0] fnode 6 : {'inputs_offsets': [3, 5, 10], 'sampling': ['keep', 'keep', 'down'], 'upsample_type': ['function', 'function', 'function']} 2024-06-24 14:35:33,796 INFO [logger.py:199] Node[0] fnode 7 : {'inputs_offsets': [4, 11], 'sampling': ['keep', 'down'], 'upsample_type': ['function', 'function']} 2024-06-24 14:35:33,797 INFO [logger.py:199] Node[0] building bifpn cell 2 2024-06-24 14:35:33,797 INFO [logger.py:199] Node[0] fnode 0 : {'inputs_offsets': [3, 4], 'sampling': ['keep', 'up'], 'upsample_type': ['function', 'function']} 2024-06-24 14:35:33,798 INFO [logger.py:199] Node[0] fnode 1 : {'inputs_offsets': [2, 5], 'sampling': ['keep', 'up'], 'upsample_type': ['function', 'function']} 2024-06-24 14:35:33,799 INFO [logger.py:199] Node[0] fnode 2 : {'inputs_offsets': [1, 6], 'sampling': ['keep', 'up'], 'upsample_type': ['function', 'function']} 2024-06-24 14:35:33,800 INFO [logger.py:199] Node[0] fnode 3 : {'inputs_offsets': [0, 7], 'sampling': ['keep', 'up'], 'upsample_type': ['function', 'function']} 2024-06-24 14:35:33,801 INFO [logger.py:199] Node[0] fnode 4 : {'inputs_offsets': [1, 7, 8], 'sampling': ['keep', 'keep', 'down'], 'upsample_type': ['function', 'function', 'function']} 2024-06-24 14:35:33,803 INFO [logger.py:199] Node[0] fnode 5 : {'inputs_offsets': [2, 6, 9], 'sampling': ['keep', 'keep', 'down'], 'upsample_type': ['function', 'function', 'function']} 2024-06-24 14:35:33,804 INFO [logger.py:199] Node[0] fnode 6 : {'inputs_offsets': [3, 5, 10], 'sampling': ['keep', 'keep', 'down'], 'upsample_type': ['function', 'function', 'function']} 2024-06-24 14:35:33,805 INFO [logger.py:199] Node[0] fnode 7 : {'inputs_offsets': [4, 11], 'sampling': ['keep', 'down'], 'upsample_type': ['function', 'function']} 2024-06-24 14:35:33,856 WARNING: GridSample module is deprecated,please use torch.nn.functional.grid_sample 2024-06-24 14:35:33,866 WARNING [pipelines.py:47] Node[0] qat_mode is set to "fuse_bn" by default 2024-06-24 14:35:33,954 WARNING [hash.py:218] Node[0] Don not found hash value in name of models/detr3d/float-checkpoint-best.pth.tar, will skip check hash... 2024-06-24 14:35:35,424 WARNING [checkpoint.py:75] Node[0] module. is not at the beginning of state dict 2024-06-24 14:35:35,458 INFO [checkpoint.py:388] Node[0] state_dict in checkpoint num: 1358 2024-06-24 14:35:35,465 INFO [checkpoint.py:389] Node[0] state_dict in model num: 1358 2024-06-24 14:35:35,466 WARNING [checkpoint.py:390] Node[0] miss_key num: 0 2024-06-24 14:35:35,466 WARNING [checkpoint.py:393] Node[0] unexpect_key num: 0 2024-06-24 14:35:35,467 INFO [converters.py:565] Node[0] Load the checkpoint successfully from ./models/detr3d/float-checkpoint-best.pth.tar 2024-06-24 14:35:38,625 INFO [loop_base.py:480] Node[0] Start Predictor loop from epoch 0, num_epochs=1 `aidisdk` dependency is not available. 2024-06-24 14:35:42,635 INFO [loop_base.py:542] Node[0] 0 / 81 2024-06-24 14:36:25,172 INFO [loop_base.py:542] Node[0] 50 / 81 2024-06-24 14:36:52,073 INFO [nuscenes_metric.py:312] Node[0] The length of self.nusc_annos is: 81 qkc:52809:52809 [0] misc/cudawrap.cc:179 NCCL WARN Failed to find CUDA library libcuda.so (NCCL_CUDA_PATH='') : libcuda.so: cannot open shared object file: No such file or directory NCCL version 2.18.5+cuda11.8 2024-06-24 14:36:52,459 INFO [nuscenes_metric.py:336] Node[0] Results writes to ./WORKSPACE/resultsdetr3d_efficientnetb3_nuscenes/results_nusc.json ====== Loading NuScenes tables for version v1.0-trainval... 23 category, 8 attribute, 4 visibility, 64386 instance, 12 sensor, 10200 calibrated_sensor, 2631083 ego_pose, 68 log, 850 scene, 34149 sample, 2631083 sample_data, 1166187 sample_annotation, 4 map, Done loading in 55.792 seconds. ====== Reverse indexing ... Done reverse indexing in 78.0 seconds. ====== Initializing nuScenes detection evaluation Loaded results from ./WORKSPACE/resultsdetr3d_efficientnetb3_nuscenes/results_nusc.json. Found detections for 81 samples. Loading annotations for val split from nuScenes version: v1.0-trainval 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████| 6019/6019 [00:05<00:00, 1029.29it/s] Loaded ground truth annotations for 6019 samples. 2024-06-24 14:40:01,965 ERROR [ddp_trainer.py:463] Node[0] Traceback (most recent call last): File "/home/qinyonghang/.local/lib/python3.10/site-packages/hat/engine/ddp_trainer.py", line 457, in _with_exception fn(*args) File "/mnt/g/code/j6/open_explorer/samples/ai_toolchain/horizon_model_train_sample/scripts/tools/predict.py", line 152, in predict_entrance predictor.fit() File "/home/qinyonghang/miniconda3/envs/j6/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/home/qinyonghang/.local/lib/python3.10/site-packages/hat/engine/predictor.py", line 117, in fit super().fit() File "/home/qinyonghang/.local/lib/python3.10/site-packages/hat/engine/loop_base.py", line 643, in fit self.on_epoch_end( File "/home/qinyonghang/.local/lib/python3.10/site-packages/hat/engine/loop_base.py", line 449, in on_epoch_end super(LoopBase, self).on_epoch_end(**kwargs) File "/home/qinyonghang/.local/lib/python3.10/site-packages/hat/engine/loop_base.py", line 114, in on_epoch_end cb.on_epoch_end(**kwargs) File "/home/qinyonghang/.local/lib/python3.10/site-packages/hat/callbacks/metric_updater.py", line 349, in on_epoch_end self._log(metrics, prefix) File "/home/qinyonghang/.local/lib/python3.10/site-packages/hat/callbacks/metric_updater.py", line 354, in _log name, value = m.get() File "/home/qinyonghang/.local/lib/python3.10/site-packages/hat/metrics/metric.py", line 198, in wrapped_func res = get(*args, **kwargs) File "/home/qinyonghang/.local/lib/python3.10/site-packages/hat/metrics/nuscenes_metric.py", line 321, in get self._get() File "/home/qinyonghang/.local/lib/python3.10/site-packages/hat/utils/distributed.py", line 61, in wrapped_fn return fn(*args, **kwargs) File "/home/qinyonghang/.local/lib/python3.10/site-packages/hat/metrics/nuscenes_metric.py", line 354, in _get nusc_eval = NuScenesEval( File "/home/qinyonghang/.local/lib/python3.10/site-packages/nuscenes/eval/detection/evaluate.py", line 84, in __init__ assert set(self.pred_boxes.sample_tokens) == set(self.gt_boxes.sample_tokens), \ AssertionError: Samples in split doesn't match samples in predictions. ^CERROR:hat.engine.ddp_trainer: Traceback (most recent call last): File "/home/qinyonghang/.local/lib/python3.10/site-packages/hat/engine/ddp_trainer.py", line 426, in launch mp.spawn( File "/home/qinyonghang/miniconda3/envs/j6/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 246, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method="spawn") File "/home/qinyonghang/miniconda3/envs/j6/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 202, in start_processes while not context.join(): File "/home/qinyonghang/miniconda3/envs/j6/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 114, in join ready = multiprocessing.connection.wait( File "/home/qinyonghang/miniconda3/envs/j6/lib/python3.10/multiprocessing/connection.py", line 931, in wait ready = selector.select(timeout) File "/home/qinyonghang/miniconda3/envs/j6/lib/python3.10/selectors.py", line 416, in select fd_event_list = self._selector.poll(timeout) KeyboardInterrupt Killed