五种并行处理策略对比调研

在处理大规模文本数据时，合理利用多进程可以显著提升处理速度。然而，并行策略的选择对性能影响巨大。本文通过一个具体的 JSONL 文件处理任务（为每行文本添加词数统计），实现并对比五种不同的多进程策略，分析其性能差异和适用场景。

所有代码均可直接复制运行，包含数据生成脚本和主处理脚本两个文件。

1. 数据生成脚本

首先，我们需要生成测试数据。以下脚本将创建data/目录，并生成指定数量和大小的.jsonl文件。

# generate_data.pyimportosimportjsonimportrandomimportshutil NUM_FILES=200# 总共生成 200 个 jsonl 文件OUTPUT_DIR="data"# 输出目录名为 inputMIN_WORDS_PER_LINE=200# 每行最少 200 个单词MAX_WORDS_PER_LINE=1000# 每行最多 1000 个单词# 极小文件：1 行# 中等文件：10 ～ 500 行# 超大文件：至少 50,000 行（可远超其他所有文件总和）SMALL_FILE_LINES=1MEDIUM_FILE_MAX_LINES=500LARGE_FILE_MIN_LINES=50000COMMON_WORDS=["the","be","to","of","and","a","in","that","have","I","it","for","not","on","with","he","as","you","do","at","this","but","his","by","from","they","we","say","her","she","or","an","will","my","one","all","would","there","their","what","so","up","out","if","about","who","get","which","go","me","when","make","can","like","time","no","just","him","know","take","people","into","year","your","good","some","could","them","see","other","than","then","now","look","only","come","its","over","think","also","back","after","use","two","how","our","work","first","well","way","even","new","want","because","any","these","give","day","most","us"]defgenerate_random_text():num_words=random.randint(MIN_WORDS_PER_LINE,MAX_WORDS_PER_LINE)words=[random.choice(COMMON_WORDS)for_inrange(num_words)]return' '.join(words)defwrite_jsonl_file(filepath,num_lines):withopen(filepath,'w',encoding='utf-8')asf:for_inrange(num_lines):line={"text":generate_random_text()}f.write(json.dumps(line,ensure_ascii=False)+'\n')defmain():ifos.path.exists(OUTPUT_DIR):shutil.rmtree(OUTPUT_DIR)os.makedirs(OUTPUT_DIR)print(f"正在重建目录: