Skip to content

F
ffm-baseline
Commit 856be02b authored by Your Name's avatar Your Name
Browse files
Options

并行处理数据转换

parent 0a4607e1
Hide whitespace changes
Inline Side-by-side
Showing with 1 addition and 1 deletion
eda/esmm/Model_pipline/train.py
View file @ 856be02b
...@@ -71,7 +71,7 @@ def input_fn(filenames, batch_size=32, num_epochs=1, perform_shuffle=False): ...@@ -71,7 +71,7 @@ def input_fn(filenames, batch_size=32, num_epochs=1, perform_shuffle=False):
return parsed, {"y": y, "z": z} return parsed, {"y": y, "z": z}
# Extract lines from input files using the Dataset API, can pass one filename or filename list # Extract lines from input files using the Dataset API, can pass one filename or filename list
dataset = tf.data.TFRecordDataset(filenames).map(_parse_fn, num_parallel_calls=10).prefetch(500000) # multi-thread pre-process then prefetch dataset = tf.data.TFRecordDataset(filenames).map(_parse_fn, num_parallel_calls=8).prefetch(500000) # multi-thread pre-process then prefetch
# Randomizes input using a window of 256 elements (read into memory) # Randomizes input using a window of 256 elements (read into memory)
if perform_shuffle: if perform_shuffle:
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment