[BUG]When fluid and alluxio configured the data set to be preheated, it was found that some files were not preheated successfully

[yizhouv5]

What is your environment(Kubernetes version, Fluid version, etc.)
K8s: v1.29.7
Containerd: 1.7.22
OS: Ubuntu 22.04.3
fluid: v1.0.2-41eefb6
alluxio/alluxio-dev:2.9.0

Describe the bug
After the fluid dataset and alluxio rumtime CR resources were created, and before the fluid PVC was mounted to the K8s container instance, the dataload CR resource was created to preheat the dataset. Occasionally, some files of the dataset could not be preheated successfully. Only the preheating failure problem is displayed in the log. Do you have specific locating methods and solutions?

What you expect to happen:
Each file should be preheated successfully

How to reproduce it
1、dataset、alluxio runtime CR yaml：
apiVersion: data.fluid.io/v1alpha1
kind: Dataset
metadata:
name: fluid-8b-preheat
spec:
mounts:

• mountPoint: local:///yunmai/llama3-ds-ckp
  name: fluid-8b-preheat
  accessModes:
  • ReadWriteMany

---

apiVersion: data.fluid.io/v1alpha1
kind: AlluxioRuntime
metadata:
name: fluid-8b-preheat
spec:
replicas: 2 # 待启动的Alluxio缓存系统Worker组件副本数。
data:
replicas: 2
tieredstore:
levels:
- mediumtype: MEM
path: /dev/shm
quota: 500Gi
high: "0.95"
low: "0.8"

2、dataload CR yaml
apiVersion: data.fluid.io/v1alpha1
kind: DataLoad
metadata:
name: fluid-8b-preheat
spec:
dataset:
name: fluid-8b-preheat
namespace: default
loadMetadata: true
target:
- path: /
replicas: 2

3、loader-job log

• alluxio fs distributedLoad --replication 2 /
  Please wait for command submission to finish..
  Submitted successfully, jobControlId = 1730198234915
  Waiting for the command to finish ...
  Get command status information below:
  Successfully loaded path /fluid-8b-preheat/llama3-ckpts/Meta-Llama-3-8B/.gitattributes
  Successfully loaded path /fluid-8b-preheat/llama3-ckpts/Meta-Llama-3-8B/LICENSE
  Successfully loaded path /fluid-8b-preheat/llama3-ckpts/Meta-Llama-3-8B/README.md
  Successfully loaded path /fluid-8b-preheat/llama3-ckpts/Meta-Llama-3-8B/USE_POLICY.md
  Successfully loaded path /fluid-8b-preheat/llama3-ckpts/Meta-Llama-3-8B/config.json
  Successfully loaded path /fluid-8b-preheat/llama3-ckpts/Meta-Llama-3-8B/configuration.json
  Successfully loaded path /fluid-8b-preheat/llama3-ckpts/Meta-Llama-3-8B/generation_config.json
  Successfully loaded path /fluid-8b-preheat/llama3-ckpts/Meta-Llama-3-8B/model-00001-of-00004.safetensors
  Successfully loaded path /fluid-8b-preheat/llama3-ckpts/Meta-Llama-3-8B/model-00002-of-00004.safetensors
  Successfully loaded path /fluid-8b-preheat/llama3-ckpts/Meta-Llama-3-8B/model-00003-of-00004.safetensors
  Successfully loaded path /fluid-8b-preheat/llama3-ckpts/Meta-Llama-3-8B/model-00004-of-00004.safetensors
  Successfully loaded path /fluid-8b-preheat/llama3-ckpts/Meta-Llama-3-8B/model.safetensors.index.json
  Successfully loaded path /fluid-8b-preheat/llama3-ckpts/Meta-Llama-3-8B/special_tokens_map.json
  Successfully loaded path /fluid-8b-preheat/llama3-ckpts/Meta-Llama-3-8B/tokenizer.json
  Successfully loaded path /fluid-8b-preheat/llama3-ckpts/Meta-Llama-3-8B/tokenizer_config.json
  Successfully loaded path /fluid-8b-preheat/llama3-ckpts/Meta-Llama-3-8B-tp2-pp4/config.json
  Successfully loaded path /fluid-8b-preheat/llama3-ckpts/Meta-Llama-3-8B-tp2-pp4/configuration.json
  Successfully loaded path /fluid-8b-preheat/llama3-ckpts/Meta-Llama-3-8B-tp2-pp4/generation_config.json
  Successfully loaded path /fluid-8b-preheat/llama3-ckpts/Meta-Llama-3-8B-tp2-pp4/latest_checkpointed_iteration.txt
  Successfully loaded path /fluid-8b-preheat/llama3-ckpts/Meta-Llama-3-8B-tp2-pp4/model.safetensors.index.json
  Successfully loaded path /fluid-8b-preheat/llama3-ckpts/Meta-Llama-3-8B-tp2-pp4/release/mp_rank_00_001/model_optim_rng.pt
  Successfully loaded path /fluid-8b-preheat/llama3-ckpts/Meta-Llama-3-8B-tp2-pp4/release/mp_rank_00_003/model_optim_rng.pt
  Successfully loaded path /fluid-8b-preheat/llama3-ckpts/Meta-Llama-3-8B-tp2-pp4/release/mp_rank_01_000/model_optim_rng.pt
  Successfully loaded path /fluid-8b-preheat/llama3-ckpts/Meta-Llama-3-8B-tp2-pp4/release/mp_rank_01_001/model_optim_rng.pt
  Successfully loaded path /fluid-8b-preheat/llama3-ckpts/Meta-Llama-3-8B-tp2-pp4/release/mp_rank_01_002/model_optim_rng.pt
  Successfully loaded path /fluid-8b-preheat/llama3-ckpts/Meta-Llama-3-8B-tp2-pp4/release/mp_rank_01_003/model_optim_rng.pt
  Successfully loaded path /fluid-8b-preheat/llama3-ckpts/Meta-Llama-3-8B-tp2-pp4/special_tokens_map.json
  Successfully loaded path /fluid-8b-preheat/llama3-ckpts/Meta-Llama-3-8B-tp2-pp4/tokenizer.json
  Successfully loaded path /fluid-8b-preheat/llama3-ckpts/Meta-Llama-3-8B-tp2-pp4/tokenizer_config.json
  Successfully loaded path /fluid-8b-preheat/llama3-datasets/wudao_llama3bpe_content_document.bin
  Successfully loaded path /fluid-8b-preheat/llama3-datasets/wudao_llama3bpe_content_document.idx
  Total completed file count is 31, failed file count is 2
  Finished running the command, jobControlId = 1730198234915
  Here are failed files:
  /fluid-8b-preheat/llama3-ckpts/Meta-Llama-3-8B-tp2-pp4/release/mp_rank_00_002/model_optim_rng.pt,
  /fluid-8b-preheat/llama3-ckpts/Meta-Llama-3-8B-tp2-pp4/release/mp_rank_00_000/model_optim_rng.pt,
  Check out ./logs/user/distributedLoad__failures.csv for full list of failed files.

real 1m13.203s
user 1m4.701s
sys 0m4.343s

• echo -e 'distributedLoad on / ends'
• (( i++ ))
  distributedLoad on / ends
• (( i<1 ))

4、preheat dataset size

du -sh *

34G llama3-ckpts
70G llama3-datasets

Additional Information