What is the current behavior?
There is no native function in snowpark-python or the snowflake-python-connector to ingest a folder of parquet files in a performant way to Snowflake. The closest is Session.write_arrow which already has all required logic.
What is the desired behavior?
Add Session.write_parquet(folder_with_parquet_files, ...) that reuses the logic from write_arrow. Refactor write_arrow to first create the parquet files and than calls Session.write_parquet
How would this improve snowflake-snowpark-python?
Many customers already have parquet files written by other engines that need to be quickly ingested into Snowflake. Avoiding the arrow step adds the most performant ingestion method to snowpark-python.
References, Other Background
What is the current behavior?
There is no native function in snowpark-python or the snowflake-python-connector to ingest a folder of parquet files in a performant way to Snowflake. The closest is Session.write_arrow which already has all required logic.
What is the desired behavior?
Add Session.write_parquet(folder_with_parquet_files, ...) that reuses the logic from write_arrow. Refactor write_arrow to first create the parquet files and than calls Session.write_parquet
How would this improve
snowflake-snowpark-python?Many customers already have parquet files written by other engines that need to be quickly ingested into Snowflake. Avoiding the arrow step adds the most performant ingestion method to snowpark-python.
References, Other Background