Major Updates

📊 Stats: 23 PRs merged, from 9 contributors

📈 Code diff: 197 files changed, with 9,415 insertions and 1,741 deletions

🧃 Juicer model release — Released Juicer-35B-A3B (ModelScope), a locally deployable data-refinement model that follows natural-language instructions for text cleaning, filtering, and semantic labeling. English and Chinese guides and the Juicer Playground provide deployment instructions and example recipes. #1060

🧮 Cluster-aware partitioning — Automatic partition counts now use live Ray cluster resources. Manual partition.size targets split data at row boundaries, including inputs with fewer blocks than partitions. #1044 #1045 #1063

✅ Config validation — Pipeline preflight catches invalid operator settings and executor/schema mismatches before processing. Reader defaults now apply consistently across execution and analysis. #1049 #1061

🔌 LiteLLM backend — New api_backend="litellm" support in prepare_api_model routes chat, embedding, and Responses requests through LiteLLM providers while retaining the existing OpenAI-compatible backend as the default. #1062

📚 Documentation refresh — Rewritten English and Chinese guides cover installation, processing, analysis, configuration, export, and the playground. Added documentation for 28 existing operators, corrected examples, and separated guide/API navigation with incremental versioned builds. #1048

🗄️ Unified remote export — Local, S3, and HDFS export share filesystem dispatch, and JSONL output now serializes Python dates and datetimes in ISO format. #1021 #1065

🖼️ Image OHEM selector — New image_ohem_selector selects high-loss image samples using a user-supplied scoring function and a top-k or ratio budget. #1038

⚡ Bounded tokenizer batches — Token-count filters limit tokenizer batch sizes to reduce peak memory on long inputs. #1040

🔧 Robustness fixes — Fixed fused-filter cache isolation, MinHash state reuse and empty inputs, deduplicator execution-mode declarations, text chunking, compressed HPO sampling, and pandas extension-dtype handling. #1059 #1043 #1051 #1057 #1046 #1055 #1047

Enhancements

• Cluster-aware auto partition counts: partition.num_of_partitions: auto derives partition counts from live Ray cluster resources. #1045
• Single Ray topology source: consolidated resource detection and removed guessed node counts and driver-local clamping in partitioned execution. #1044
• Manual partition sample targets: partition.size specifies an approximate number of rows per partition; explicit count and sample-target settings are validated as alternatives. #1063
• Pipeline preflight validation: validate operator parameters, executor compatibility, and required dataset fields before processing across default, Ray, and partitioned Ray execution. #1049
• Consistent reader defaults: dataset loading and analysis share reader configuration, with explicit load arguments taking precedence over recipe defaults and np. Exposed the existing load_jsonl_lenient and use_dag controls in YAML/CLI. #1061
• LiteLLM provider integration: chat, embedding, and Responses wrappers support provider-specific routing and credentials through the opt-in LiteLLM backend. #1062
• Unified remote filesystem dispatch: Exporter and RayExporter share path resolution for local files, S3, and HDFS. #1021
• Bounded tokenizer batches: cap token-count tokenizer batch sizes while preserving filtering behavior. #1040
• Bilingual guides and operator reference: rewrite task guides, document 28 existing operators, fix examples and links, and update guide/API navigation and versioned documentation builds. #1048
• Juicer model documentation: added English and Chinese guides for the separately announced Juicer model (ModelScope) and playground. #1060
• Maintenance forks: updated recognize-anything and transformers-stream-generator references to maintained personal forks. #1056
• Dependency and CI updates: bumped NLTK to 3.10.3 and updated checkout/setup-python action majors in the pre-commit workflow. #1066 #1067

New OPs

• Image OHEM selector: image_ohem_selector keeps the highest-loss image samples, with callable or file-based scoring functions, an optional model factory, and top-k or ratio selection. #1038

Fixed Bugs

• Fused-filter context cache isolation: cache keys now include the text key and tokenizer configuration, preventing statistics from being reused across different columns or tokenization settings. #1059
• MinHash stale hash tables: reset hash tables between process() calls so an operator instance can process independent datasets without carrying over previous state. #1043
• MinHash empty token sets: return an all-maximum signature instead of crashing on an empty reduction; zero-token samples share a signature and are deduplicated together. #1051
• Deduplicator execution modes: correctly declare local-only and Ray support so incompatible executor/operator combinations are rejected. #1057
• JSONL date serialization: Python date and datetime values are serialized with isoformat() instead of raising TypeError. #1065
• Extensionless export paths: report how to provide a file suffix or explicit export_type when the output path has no extension. #1053
• Empty text chunks: recursively_chunk no longer emits empty fragments. #1046
• HPO gzip JSONL sampling: read compressed statistics through gzip and clean up temporary samples even when analysis fails. #1055
• pandas extension dtypes: use pandas' numeric-dtype predicate for analysis compatibility. #1047
• Ray JSON reader options: forward configured read options consistently, preserve disabled reader threading by default, and fix the older-PyArrow fallback. #1061
• Global configuration consistency: remove unused intermediate_storage.*, preserve_intermediate_data, resource_optimization.auto_configure, max_log_size_mb, and backup_count declarations; remove the stale add_suffix example; reject unsupported checkpoint strategies and nonpositive intervals. #1061
• Partition-size configuration: connect the sample target to row-based splitting, handle single-partition and remainder cases, and remove the unused max_partition_size_mb field. The legacy partition_size remains accepted with a deprecation warning; partition.target_size_mb is a planning estimate rather than a hard size limit. #1063
• Import-safe download and HPO tooling: correct package imports in the arXiv downloader and HPO entry point; importing the W&B HPO module no longer starts a sweep. #1048
• Regression test discovery and assertions: remove the preflight test's import-path override and align MinHash/Ray regression cases with current behavior, including runtime-environment restoration after failed retries.

Acknowledgements

• @nofear-spec contributed the image OHEM selector for hard-example mining. #1038
• @macroguo-ghy contributed bounded tokenizer batching to reduce peak memory. #1040
• @mameikagou fixed gzip JSONL statistics sampling for HPO. #1055
• @arkrolin fixed fused-filter context cache isolation. #1059
• @prodmanpd contributed LiteLLM provider integration. #1062
• @JohnGiorgi fixed date and datetime serialization in JSONL exports. #1065
• @katsugtgz updated NLTK and the pre-commit workflow actions. #1066 #1067

New Contributors

• @nofear-spec made their first contribution in #1038
• @mameikagou made their first contribution in #1055
• @arkrolin made their first contribution in #1059
• @prodmanpd made their first contribution in #1062
• @katsugtgz made their first contribution in #1066

Full Changelog: v1.5.5...v1.6.0

Major Updates

📊 Stats: 18 PRs merged, from 5 contributors

📈 Code diff: 84 files changed, with 11,991 insertions and 389 deletions

🧩 External operator plugins — Support loading external operators via Python entry points, allowing third-party OPs to be shipped as independent pip packages and used with zero extra config. #1026

🗄️ HDFS protocol support — Data-Juicer can now read datasets from and export processed results to HDFS (hdfs://), consistent with the existing S3 remote-storage workflow. #1014

⚡ Leaner & faster Ray Data — Removed eager count()/columns()/schema() actions that triggered premature Ray Dataset execution, replaced the private get_compute_strategy helper with the public TaskPoolStrategy/ActorPoolStrategy APIs, and made PartitionedRayExecutor run partitions concurrently. #1022 #1024 #1025 #1032 #1036

🌐 Elastic multi-node sharding — Available as a runnable reference workflow (scripts + sample configs under demos/elastic_sharding/) that pre-splits a large JSONL dataset into shards and processes them elastically across multiple nodes with a node-local Data-Juicer Ray executor, including retries, stale-claim recovery, status inspection, and ordered merge. #1015

📊 RayAnalyzer for distributed analysis — New RayAnalyzer computes filter stats via Ray map_batches and aggregates overall statistics (count/mean/std/min/max) using Ray native aggregation operators, without pandas materialization. It is the distributed counterpart to the local Analyzer. #1016

Enhancements

• External operator plugins via entry points: third-party OPs can be shipped as independent pip packages and are loaded into the global OPERATORS registry at import data_juicer.ops time. #1026
• HDFS protocol support for dataset loading and exporting. #1014
• Public Ray compute strategies: replaced the private get_compute_strategy helper with the public TaskPoolStrategy/ActorPoolStrategy APIs. #1024
• Reduced eager Ray dataset actions: removed premature count(), switched to take(k)/take_all(), and avoided repeated columns()/schema() calls. #1025
• Elastic multi-node sharding: available as a runnable reference workflow (scripts + sample configs under demos/elastic_sharding/) that pre-splits a large JSONL dataset into shards and processes them elastically across multiple nodes with a node-local Data-Juicer Ray executor (retries, stale-claim recovery, status inspection, and ordered merge). #1015
• RayAnalyzer for distributed data analysis: computes filter stats via Ray map_batches and aggregates overall statistics without pandas materialization. #1016
• Membership operators (in/not in) in general_field_filter for expressions like lang in ['en', 'zh']. #1000
• Streamed n-gram frequency counting: count character and word n-grams as they are generated instead of retaining an occurrence-sized intermediate list, cutting peak memory in character_repetition_filter and word_repetition_filter while preserving ratios, caching, and filtering decisions. #1029
• Bounded MinHash permutation workspace: reduce token-by-permutation matrices in 8 MiB blocks and fold exact minima in document_minhash_deduplicator, cutting RSS from 768.0 to 271.9 MiB at 4x while preserving byte signatures. #1035
• Test refactor & coverage: enhanced tests for base_op, job utils, and config. #1018

Fixed Bugs

• Concurrent partitioned Ray execution: PartitionedRayExecutor now drives partitions concurrently via a driver-side ThreadPoolExecutor instead of serially, preserving partition order. #1022
• Elastic actor pool concurrency: ActorPoolStrategy now honors (min, max) num_proc ranges instead of collapsing them to a fixed size. #1032
• Partition checkpoint preservation: partition checkpoints now survive Ray block-layout changes, so resumed runs no longer reprocess completed partitions. #1036
• text_chunk_mapper visible delimiters: split_pattern no longer emits visible delimiters (e.g. _@@@_) as standalone chunks. #999
• calibrate_response_mapper output pattern: now honors output_pattern for regex extraction instead of ignoring it. #1017
• image_diffusion_mapper null captions: handles missing or empty caption fields gracefully instead of crashing. #1020
• Silent no-op on empty OP list: DefaultExecutor and RayExecutor now warn when the process operator list is empty. #1020
• pandas 3 incompatibility with Ray Data: pinned pandas<3 in the distributed extra to avoid AttributeError during JSON export. #1023
• pyarrow>=17 compatibility: bumped fsspec accordingly. #1020
• Python 3.13+ dependency resolution: removed the numpy<2.0 upper bound (blocked tf-keras/ml-dtypes needing numpy>=2.1.0) and upgraded label-studio to >=1.23.0. #1027

Acknowledgements

• @LiuGuH contributed HDFS protocol support for dataset loading and exporting. #1014
• @macroguo-ghy contributed streamed n-gram frequency counting and a bounded MinHash permutation workspace to cut peak memory. #1029 #1035

New Contributors

• @LiuGuH made their first contribution in #1014

Full Changelog: v1.5.4...v1.5.5

Major Updates

📊 Stats: 10 PRs merged, from 6 contributors

📈 Code diff: 125 files changed, with 11,287 insertions and 766 deletions

🧑‍🤝‍🧑 New OPs — Added 9 human-centric video understanding operators (human track extraction, active-speaker detection, audio ASR, speech emotion & age/gender detection, face demographic & attribute/emotion captioning, face-ratio filtering) for building HumanVBench-style pipelines. (These OPs need third-party patches/models under thirdparty/humanvbench_models/ and require a source install for now.) #938 #1013

⚡ Batch-local stage fusion — New FusedSequentialBatchOp runs a list of batch-local sub-operators sequentially inside a single dataset map stage, reducing scheduler/stage overhead for mapper/filter chains. Wired through DefaultExecutor, RayExecutor, and PartitionedRayExecutor. #1004

New OPs

• 9 HumanVBench human-centric video OPs — 8 mappers (video_human_tracks_extraction_mapper, video_active_speaker_detect_mapper, video_audio_ASR_mapper, video_audio_speech_emotion_mapper, video_audio_detect_age_gender_mapper, video_human_tracks_face_demographic_mapper, video_captioning_face_attribute_emotion_mapper, video_captioning_from_human_tracks_mapper) + 1 filter (video_face_ratio_filter). #938 #1013
• fused_sequential_batch_op (FusedSequentialBatchOp). #1004

Fixed Bugs

• HumanVBench ops runtime compatibility: scenedetect open_video API migration, SenseVoiceSmall model resolution via ModelScope, output-directory creation on init, and declared runtime deps. #1013
• Ray deduplicator shared state: share dedup backend state across map_batches tasks, reuse actor handles, materialize stats for stateful backends. #978
• Ray checkpoint relative path: resolve work_dir/checkpoint_dir to absolute paths in PartitionedRayExecutor. #979
• clean_html_mapper on null text: no longer errors on null/None text. #1001
• Boolean stat columns in ColumnWiseAnalysis: dj-analyze no longer crashes on boolean stat columns. #1003
• ARM64 (aarch64) install unblocked: precise decord/torchcodec platform markers restrict them to platforms with pre-built wheels (Linux x86_64, Windows x86_64, macOS arm64). #998

Enhancements

• Test coverage & cleanup: expanded utility/model-handling tests, plus assorted cleanup. #1002 #1007

Acknowledgements

• @macroguo-ghy fixed Ray deduplicator shared-state handling. #978
• @wenhaozhao011-cmd fixed boolean stat column handling in ColumnWiseAnalysis. #1003
• @bglglzd contributed code/documentation cleanup. #1007

New Contributors

• @bglglzd made their first contribution in #1007

Full Changelog: v1.5.3...v1.5.4

Major Updates

📊 Stats: 14 PRs merged, from 8 contributors

📈 Code diff: 172 files changed, with 19,085 insertions and 2,144 deletions

🤖 VLA ops enhancements: Expanded embodied-AI / Vision-Language-Action processing capabilities with 10+ new and renamed operators — including new camera calibration methods (DeepCalib, DroidCalib, MoGe), atomic action segmentation, hand action computation & motion smoothing, clip reassembly, trajectory overlay, and LeRobot export — plus a complete VLA pipeline demo for ego-hand action annotation. #931

🔄 Ray repartition pipeline: A new ray_repartition_pipeline enables dataset-level block repartitioning in Ray mode, giving users fine-grained control over data distribution across workers. #985

⚡ Scalable Ray Data reads: Wired override_num_blocks through the full call chain, allowing users to control Ray Data's block parallelism via CLI — essential for processing PB-scale datasets without overwhelming the scheduler. #984

🧪 Test coverage expansion: Added 409 new test cases across 18 test files covering utils, ops, format, config, download, and pipeline DAG modules. #990

New OPs

• export_to_lerobot_mapper: Exports processed data into the LeRobot dataset format for downstream robot learning. #931
• video_atomic_action_segment_mapper: Segments videos into atomic actions for fine-grained action annotation. #931
• video_camera_calibration_deepcalib_mapper (renamed from video_camera_calibration_static_deepcalib_mapper): Computes camera intrinsics and FOV using DeepCalib. #931
• video_camera_calibration_droidcalib_mapper: Computes camera intrinsics and FOV using DroidCalib. #931
• video_camera_calibration_moge_mapper (renamed from video_camera_calibration_static_moge_mapper): Computes camera intrinsics and FOV using MoGe-2. #931
• video_camera_pose_megasam_mapper (renamed from video_camera_pose_mapper): Extracts camera poses using MegaSaM and MoGe-2. #931
• video_clip_reassembly_mapper: Reassembles video clips for flexible clip-level data organization. #931
• video_hand_action_compute_mapper: Computes hand action data from video for manipulation tasks. #931
• video_hand_motion_smooth_mapper: Smooths hand motion trajectories for cleaner action signals. #931
• video_trajectory_overlay_mapper: Overlays trajectory visualizations onto video frames for debugging and presentation. #931
• ray_repartition_pipeline: A Ray-only pipeline for dataset-level block repartitioning, registered in config_all.yaml and operator docs. #985

Enhancements

• override_num_blocks CLI argument for Ray Data: Previously implemented only at the lowest layer (read_json_stream()), this parameter is now wired through the full call chain, making it accessible via CLI for controlling block parallelism on very large datasets. #984
• num_proc handling for vllm and Ray mode: TextTaggingByPromptMapper was unconditionally setting num_proc = 1, which broke parallelism in Ray mode. Now properly respects the configured value. #973

Fixed Bugs

• JSONStreamDatasource schema mismatch across batches: The first batch's inferred schema was locked and reused for all subsequent batches. When an early batch inferred a field as null and a later batch introduced a concrete type (e.g., string), the forced cast failed with ArrowInvalid. Schema is now unified across batches. #972
• OP env LATEST strategy returning unpinned version: The conflict resolution strategy incorrectly fell back to an unpinned version when the union of two conflicting specifiers contained ranges without an upper bound (e.g., numpy>=2.0 vs numpy<1.5). Now correctly resolves to a pinned version. #992
• FUSE-safe rmtree fallback missing in PartitionedRayExecutor: PR #943 fixed shutil.rmtree() failures on FUSE-mounted OSS buckets in RayExecutor, but the same pattern was missing in ray_executor_partitioned.py. All three rmtree sites now have the fallback. #988
• Deprecated model names in tests, demos, and docs: Replaced deprecated model names (e.g., qwen2.5-72b-instruct, qwen2.5-vl-3b-instruct) with available alternatives across test files, demo configs, and docstrings. #994

Acknowledgements

• @macroguo-ghy contributed the new ray_repartition_pipeline for Ray mode. #985
• @ArdalanM fixed num_proc handling for vllm and Ray mode. #973
• @CodyQin fixed the OP env LATEST strategy version resolution bug. #992
• @wenhaozhao011-cmd added FUSE-safe rmtree fallback to PartitionedRayExecutor. #988

New Contributors

• @ArdalanM made their first contribution in #973
• @CodyQin made their first contribution in #992
• @wenhaozhao011-cmd made their first contribution in #988

Full Changelog: v1.5.2...v1.5.3

Major Updates

📊 Stats: 15 PRs merged, from 11 contributors

📈 Code diff: 217 files changed, with 18,972 insertions and 1,685 deletions

🧠 Semantic OPs MVP shipped: Introduced a new semantic operations framework with extract and condition filter capabilities, laying the groundwork for join/aggregation/top-k in future releases. #948

🤖 Agent interaction quality evaluation: New OPs and recipe for assessing interaction quality, with bad-case HTML report generation, and robust JSONL / HuggingFace meta loading support. #957

📄 Cross-document line-level deduplication: A new DocumentLineDeduplicator enables deduplication at the line level across multiple documents — useful for removing boilerplate or repeated content shared between files. #961

⚡ Lighter default installation: Reduced the size of default dependencies to significantly speed up first-time installation. #959

New OPs

• DocumentLineDeduplicator: Performs cross-document line-level deduplication — ideal for removing shared boilerplate text, repeated headers/footers, or duplicated content across large corpora. #961
• semantic_extract_mapper & semantic_condition_filter (MVP): Extract structured fields from unstructured text and filter based on semantic conditions, powered by the new semantic ops framework. #948
• Interaction quality OPs (via dj-agents): Evaluate the quality of agent interactions with dedicated scoring operators and a recipe that generates HTML bad-case reports for human review. #957

Enhancements

• O(n²) → O(n) in FrequencySpecifiedFieldSelector: Replaced repeated sum() accumulation with itertools.chain, eliminating quadratic behavior on large batches. #975
• Exclude work_dir from fingerprint hashing: Cache fingerprints no longer change when work_dir differs, preventing unnecessary recomputation while preserving correct pickling behavior. #967
• Reduce default dependency size: Trimmed the default install footprint so new users can get started faster without pulling in heavyweight optional packages. #959
• Replace bs4 stub with beautifulsoup4: Switched to the canonical package name to prevent installation conflicts and deprecation warnings. #977

Fixed Bugs

• Invalid max_new_tokens injected into API chat completions: The parameter was being incorrectly passed to chat completion API calls that don't support it, causing request failures. Now properly excluded. #983
• initialize_ray crash without config: Ray initialization failed when no config was provided. Added proper fallback handling. #981
• exit(1) replaced with raise in dataset processing: Using exit() in a library context kills the entire process. Errors now raise proper exceptions for library-safe error handling. #974
• PyArrow 20.0.0+ batch JSON reading compatibility: Fixed an issue where open_json batch reading broke under PyArrow 20+. #942
• text_keys not propagated to ops in service mode: The DJ service was not passing text_keys through to operators via get_init_configs, causing ops to use wrong keys. #960
• Model utils sampling params & API client initialization: Fixed incorrect sampling parameter handling and optimized API client setup in model_utils.py. #962
• Aesthetics-predictor normalization check too strict: The substring match for model normalization detection now supports local model paths, not just HuggingFace hub identifiers. #946
• Temp dir removal failure in RayExecutor: Added fallback handling when temporary directory cleanup fails (e.g., due to permission issues or concurrent access). #943

Acknowledgements

• @dubin555 delivered key performance optimizations and library-safety improvements. #974 #975
• @amanyara contributed the new DocumentLineDeduplicator. #961
• @ideal added robustness to RayExecutor temp dir cleanup. #943
• @HunterLine fixed PyArrow 20+ batch JSON reading compatibility. #942
• @justinwolfington corrected the beautifulsoup4 package dependency. #977
• @macroguo-ghy fixed Ray initialization without config. #981

New Contributors

• @justinwolfington made their first contribution in #977
• @macroguo-ghy made their first contribution in #981
• @amanyara made their first contribution in #961
• @ideal made their first contribution in #943

Full Changelog: v1.5.1...v1.5.2

Major Updates

• 📊 Stats: 13 PRs merged, from 7 contributors
• 📄 Two new LaTeX-focused mapper OPs shipped, extending data-juicer's document processing capabilities to handle .tex archives and figure contexts.
• 🗜️ Compressed dataset format support: json[l].gz files can now be loaded directly, and Ray datasets gain proper support for reading compressed JSON files.
• 📚 New documentation added covering cache, export, and tracing workflows to help users better understand and debug data processing pipelines.
• 🤖 Major refactor and upgrade of data-juicer-agents completed: The project architecture and CLI/session capabilities were comprehensively redesigned for better maintainability and extensibility. See date-juicer-agents for more details.

New OPs

• latex_merge_tex_mapper: Got a bunch of .tex files packed in an archive? This OP automatically extracts and merges them into a single unified LaTeX document, making it much easier to process multi-file LaTeX projects. #932
• latex_figure_context_extractor_mapper: Extracts figure-related context (e.g., captions, surrounding paragraphs) from LaTeX source files, so you can build richer multimodal datasets from academic papers. #923

Enhancements

• Load dataset with extra kwargs: You can now pass arbitrary extra arguments to datasets.load_dataset() via the new load_dataset_kwargs config field — handy for datasets that need non-standard loading options. #922
• Custom tokenizer in RemoveRepeatSentencesMapper: The mapper now accepts a custom tokenizer, so you're no longer stuck with the default sentence splitter — great for non-English text or domain-specific tokenization needs. #925
• Compressed JSON support: Added support for reading json[l].gz files directly, and fixed Ray datasets to properly handle compressed JSON — no more manual decompression before feeding data in. #919
• Faster TokenNumFilter with batch tokenization: Instead of tokenizing one sample at a time, TokenNumFilter now processes the whole batch in one shot, significantly speeding up token-count-based filtering. #929
• Cache redundant sum() calls in repetition filters: Repetition filters were calling sum() multiple times on the same data. These results are now cached, saving unnecessary computation on large batches. #924
• New docs: cache, export, and tracing: Added dedicated documentation pages explaining how data-juicer handles caching, result exporting, and execution tracing — a much-needed addition for debugging complex pipelines. #935
• Enhanced op_search with BM25/Regex & MCP Server upgrade: Added BM25 and regex search modes to op_search (no longer requiring dj-agents), and expanded the MCP server with four new tools covering op search, dataset analysis, config schema retrieval, and dataset loading strategy discovery. #937

Fixed Bugs

• Wrong cache key in ImageFaceCountFilter: The filter was using an incorrect key when reading from cache, causing it to miss cached results and redo redundant work. Now fixed. #921
• GeneralFusedOP silently dropping Mapper results: When running a fused pipeline, Mapper outputs were being discarded instead of passed downstream. This was a silent data loss bug — now properly fixed. #928
• Shared _default_kwargs mutation polluting other OP instances: Operator instances were accidentally sharing a mutable default kwargs dict, meaning modifying one OP's config could inadvertently affect other instances. Each instance now gets its own copy. #926
• NlpaugEnMapper only augmenting the first sample in a batch: Due to a bug in the batching logic, text augmentation was only being applied to the very first sample, leaving the rest untouched. All samples in a batch are now correctly augmented. #927

Acknowledgements

• @JohnGiorgi contributed three impactful improvements: load_dataset_kwargs support, custom tokenizer in RemoveRepeatSentencesMapper, and batch tokenization optimization in TokenNumFilter. #922 #925 #929
• @dubin555 squashed multiple operator bugs and added performance optimizations across filters and the fused pipeline. #921 #924 #926 #927 #928
• @leeyyi and @liyuyi-2001 made their first contributions with two brand-new LaTeX OPs. #923 #932
• @HunterLine added compressed JSON dataset support. #919

Full Changelog: v1.5.0...v1.5.1

Major Updates

• 📊 Stats: 244 files changed with 22,394 additions and 2,053 deletions, from 12 contributors
• 🗂️ New partitioned ray executor: #748
  • Support data partitioning, checkpointing, event logging in ray mode.
  • Improved fault tolerence, extensibility, observability, flexibility, and processing performance.
• 🤖 New OPs for embodied AI: improved processing capability to handle camera-view videos.
• 🧩 Support OP-level isolated environment maintaining in ray mode to help resolve the dependency confliction issue from different OPs. #892
  • Allow to merge possible environments from different OPs that share common dependencies in different strategies and reuse the created environments.
  • Based on ray runtime environment.

New OPs

• video_camera_calibration_static_deepcalib_mapper: Compute the camera intrinsics and field of view (FOV) for a static camera using DeepCalib. #871
• video_camera_calibration_static_moge_mapper: Compute the camera intrinsics and field of view (FOV) for a static camera using Moge-2. #871
• video_undistort_mapper: Undistort raw videos with corresponding camera intrinsics and distortion coefficients. #871
• video_hand_reconstruction_hawor_mapper: Use HaWoR and MoGe-2 for hand reconstruction. #893
• video_camera_pose_mapper: Extract camera poses with MegaSaM and MoGe-2. #894

Enhancements

• Allow batch inference for image_captioning_mapper to improve processing performance. #901
• Optimize the logics of a branch by avoiding unnecessary function calls. #903 '
• Refactor Operator Search and Metadata Extraction for Enhanced Accuracy. #889
• Allow to return meta infos only for extract_keyframes func and remove the sample info in error logs to reduce the size of logs. #904
• Reduce the memory usage in convert_to_absolute_paths func by iterating only over the specified columns. #907
• Reorganize the main readme and update the tutorials in the playground to the latest version. #908
• Optimize issue templates: emphasize English usage and add Q&A Copilot check. #912
• Convert abs path for dataset in object store. #913

Fixed Bugs

• Fix the bug to make minhash deduplicator be able to trace all duplicate items. #906
• Fix the "multiple values for num_proc" bug in TextFormmater. #905
• Fix the homepage rendering issue and remove outdated OP docs. #910
• Fix several bugs in test stability and robustness. #918

Acknowledgement

• @dubin555 helps to improve the processing performance of some OPs and funcs. #901 #903
• @HunterLine helps to fix a bug in minhash deduplicator to trace all duplicate items. #906

New Contributors

• @HunterLine made their first contribution in #906
• @Dludora made their first contribution in #907

All Contributors

@HYLcool @dubin555 @claude @Qirui-jiao @cmgzn @Cathy0908 @Dludora @yxdyc @gemini-code-assist @HunterLine @ext.wanghao204 @cyruszhang

Full Changelog: v1.4.6...v1.5.0

Major Updates

• 🤖 Our Q&A copilot is introduced to resolve questions from users. Now the robot is available in the docs, DingTalk group, Discord, etc. #891
• 🎬 I/O for video bytes: support bytes reading/storing for videos. #882
• 🫆 Tracer for ray mode: now the tracer supports to trace changed samples in ray mode. #885

Enhancements

• Prepare a new dockerfile for use case of embodied AI, and update the cuda/system/... versions of the basic docker image. #887
• Add Copilot News & Refined DingTalk link/QR code & Discord link/QR code in the docs. #891
• Convert the word retrieval from lists to sets to speed up two OPs. #890
• Add a new workflow to automatically fetch the traffic report from github insigts. #899 #900

Fixed Bugs

• Fix TypeError when using field_types in YAML config for RequiredFieldsValidator. #886
• Replace the deprecated concurrency parameter with compute parameter in the ray.data.Dataset.map_batches() call. #888
• Prevent divide-by-zero in calculate_ray_np when Ray cluster not ready. #864
• Add thread limiting for multi-process workloads to prevent over-subscription. #877
• Fix the bug where the unittest of standalone mode could be stuck. #896
• Update several out-of-date links in the docs. #898

Acknowledgement

• @dubin555 helps to fix several bugs and enhance the processing performance for some OPs. #886 #890
• @xyuzh helps to update the ray usage to the latest version in some OPs, fix some bugs and optimize the parallel strategies. #888 #864
• @XinyuLiu1999 helps to fix a bug of over-subscription on multi-process workloads. #877

Full Changelog: v1.4.5...v1.4.6

Major Updates

• Add several new OPs for embodied-AI.
• Upgrade to the documentation system: #842
  • transition the documentation generation and deployment to a unified Sphinx-based framework.
  • architectures, styles are maintained as an isolated repo. It will be pulled before building the docs of each sub-repo.

New OPs

Mapper

• video_captioning_from_vlm_mapper: generate video captions from latest VLMs (e.g., Qwen3-VL). #820
• video_object_segmenting_mapper: perform text-guided semantic segmentation of valid objects throughout the video (using YOLOE and SAM2), with support for saving segmentation visualization results. #801
• video_depth_estimation_mapper: perform depth estimation on the video, with support for saving both visualization results and point cloud data. #801
• image_mmpose_mapper: perform human keypoint detection inference using MMPose models. #800
• image_tagging_vlm_mapper: generates image tags with VLMs. #800
• image_sam_3d_body_mapper: perform single-image full-body 3D human mesh recovery (HMR) with the promptable model SAM 3D Body (3DB). #843
• s3_download_file_mapper: download files from S3 to local files or load them into memory. #839
• s3_upload_file_mapper: upload local files to S3 and update paths to S3 URLs. #839

Filter

• text_tagging_by_prompt_mapper: generate text tags using prompt with LLM. #408
• image_subplot_filter: detect and remove samples with images containing subplots. #840 #822
• video_motion_score_ptlflow_filter: a new motion score filter where the optical flows are computed by the ptlflow library. #824

Deduplicator

• document_minhash_deduplicator_with_uid: a more robust version of document_minhash_deduplicator for datasets with unique ID for each sample. #832 #677
• ray_bts_minhash_deduplicator_with_uid: a more robust version of ray_bts_minhash_deduplicator for datasets with unique ID for each sample. #832 #677
• ray_bts_minhash_cpp_deduplicator: enhance the performance of the basic BTS MinHash deduplicator by migrating its computationally intensive parts to C++ and Cython. #851

Pipeline

A new type of OP, which allows combine multiple OPs into one pipeline, or integrating a whole pipeline that is not easy to split into multiple atomic OPs. #835

• ray_vllm_engine_pipeline: basic OP for making use of the vLLM engine of Ray.
• llm_ray_vllm_engine_pipeline: generate response with LLMs using vLLM engine on Ray.
• vlm_ray_vllm_engine_pipeline: generate response with VLMs using vLLM engine on Ray.

Enhancements

• Several major dependencies of data-juicer are updated to the (nearly) latest version. #820
• Rename and align some core arguments of base OP about resource allocation to the ones in Ray. #837
• Refine the badges on the homepage to enhance user experience. #841
• Support to specify extra arguments for ffmpeg for some video OPs. #847
• Update Used by & Valuable Feedback from list to add new customers/users of Data-Juicer. #852
• Improve Ray-based deduplicators by lazily initializing actors, allowing the cluster to autoscale before actors consume resources. #855
• Improve the RayS3DataLoadStrategy class to provide better format detection, more informative logging, and support for loading multiple files from S3 directories in Ray mode. #860
• Support OpenAI Reponses API. #856
• Use consistent key naming and allow to include extra fields in the tracer output with trace_keys arg. #873 #874
• Support bytes data as input and add the auto_op_parallelism parameter to control whether to enable automatic calculation of OP parallelism. #867
• Support to save optical flows computed in the OP. #824
• Update the base image of official data-juicer docker image to cuda12.6.3 and ubuntu 24.04; update the python version to py311; install several embodied-ai-related packages. #881
• Add memory reservation parameters for Ray minhash deduplication to allow users to reserve memory for actors and tasks. #863

Fixed Bugs

• Fix the issue where export_aws_credentials cannot be read properly in RayExecutor when using S3 export paths. #834
• Change the logger level of import timing from info to debug. #859
• Cache the dataset columns once at the start of process() and pass the cached set through the operator pipeline to fix the issue where Ray's Dataset.columns() breaks streaming pipelines when called repeatedly during operator processing. #854
• Fix out-of-date URLs of PAI demos. #865
• Limit versions of several dependencies to avoid new issues from their latest versions. #876
• Use mount disk to avoid "No space" error on the / path when building docs. #857
• Use persist to avoid OOM in distributed deduplication of pyspark version. #836 #586
• Fix the issue where the number of OPs change can not trigger the OP doc building hook. #824

Acknowledgement

• @kyo-tom helps fix several bugs and enhance the functions of I/O, and implement 2 new OPs for s3. #834 #860 #839
• @xyuzh helps fix and enhance the core parts of ray distributed mode according to the professional techs from Ray repo. #859 #855 #854 #863
• @JohnGiorgi helps to support new type of OpenAI API and enhance the tracer. #856 #873 #874
• @coolderli helps to optimize the spark distributed deduplication to avoid OOM. #586
• @ZiyiTsang helps to implement a new OP image_subplot_filter. #822

Full Changelog: v1.4.4...v1.4.5

Major Updates

• 🎉 Update NeurIPS 2025 News: our Data-Juicer 2.0 paper is accepted as a NeurIPS'25 Spotlight (top 3.1% of all submissions)! And our two other works are also accepted by NeurIPS'25. #788
• 🧩 The sandbox component, data-juicer recipes, and data-juicer agents have been officially split from the main repository as data-juicer-sandbox/hub/agents respectively, to enable independent development and faster iteration. #817 #827 #830
• 🤝 S3 I/O support: Added S3 support in data loader and exporter for seamless cloud storage integration. #806

New OPs

• detect_main_character_mapper: Extract all main character names based on the given image and its caption. #795
• detect_character_locations_mapper: Given an image and a list of main character names, extract the bounding boxes for each present character. (YOLOE + MLLM) #795
• detect_character_attributes_mapper: Takes an image, a caption, and main character names as input to extract the characters' attributes. #795
• vggt_mapper: Input a video of a single scene, and use VGGT to extract information including Camera Pose, Depth Maps, Point Maps, and 3D Point Tracks. #804
• video_whole_body_pose_estimation_mapper : Input a video containing people, and use the DWPose model to extract the body, hand, feet, and face keypoints of the human subjects in the video, i.e., 2D Whole-body Pose Estimation. #812
• video_hand_reconstruction_mapper : Use the WiLoR model for hand localization and reconstruction. #818

Enhancements

• Enhanced documentation for operator details, significantly expanding coverage of effect demonstrations and usage examples, and improved homepage styling for better readability. #778 #819
• Added notebook detection and auto-redirect in logger setup for better user experience in Jupyter environments. #790
• Optimized the build_op_doc hook for more reliable documentation generation. #794
• Improved auto num_proc calculation in Ray mode for better resource utilization across operators. #789 #825
• Enabled support for videos and audios in WebDataset I/O, expanding multimodal data handling capabilities. #803
• Updated repository URLs and links across the project for consistency and correctness. #805
• Added support for FFmpeg and Decord backends in video data processing, improving flexibility and performance. #826 #829
• Added an MCP server CLI entry point to facilitate modular service deployment and upodate MCP documentation. #798

Fixed Bugs

• Fixed the Auto Prompt pipeline in sandbox to restore correct prompt generation behavior. #791
• Fixed a Ray connection error by properly passing the config parameter through resource utility functions. #808
• Fixed several CUDA-based operators to use internal resource monitor. #809
• Fixed custom op module loading issues and optimized video_extract_frames_mapper for saving extracted frames. #803
• Reset num_proc for vLLM and set default batch_size to 10 for CUDA operators to improve stability. #814
• Fixed Sphinx autodoc compatibility issue in the SpecialTokens metaclass to restore documentation build. #816
• Resolved a bug in trace_filter by excluding the __dj_stats__ column during dataset comparison. #828
• Fix several typos in video_split_by_scene_mapper. #744

Acknowledgement

• @kyo-tom helps to fix the ray connection error in #808
• @liuyuhanalex helps to fix several small typos in #744

Full Changelog: v1.4.3...v1.4.4