================================================================================================ Benchmark to measure CSV read/write performance ================================================================================================ OpenJDK 64-Bit Server VM 11.0.10+9-LTS on Linux 5.4.0-1043-azure Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz Parsing quoted values: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------ One quoted string 35546 35913 327 0.0 710924.3 1.0X OpenJDK 64-Bit Server VM 11.0.10+9-LTS on Linux 5.4.0-1043-azure Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz Wide rows with 1000 columns: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------ Select 1000 columns 97613 98458 1123 0.0 97613.1 1.0X Select 100 columns 42208 42598 374 0.0 42208.4 2.3X Select one column 37602 38233 547 0.0 37601.6 2.6X count() 6343 6432 153 0.2 6343.4 15.4X Select 100 columns, one bad input field 65577 66403 829 0.0 65577.2 1.5X Select 100 columns, corrupt record field 79049 79718 608 0.0 79048.6 1.2X OpenJDK 64-Bit Server VM 11.0.10+9-LTS on Linux 5.4.0-1043-azure Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz Count a dataset with 10 columns: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------ Select 10 columns + count() 17730 18004 321 0.6 1773.0 1.0X Select 1 column + count() 12627 12858 292 0.8 1262.7 1.4X count() 4329 4425 130 2.3 432.9 4.1X OpenJDK 64-Bit Server VM 11.0.10+9-LTS on Linux 5.4.0-1043-azure Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz Write dates and timestamps: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------ Create a dataset of timestamps 1539 1582 47 6.5 153.9 1.0X to_csv(timestamp) 12782 12980 192 0.8 1278.2 0.1X write timestamps to files 10122 10253 170 1.0 1012.2 0.2X Create a dataset of dates 1646 1765 111 6.1 164.6 0.9X to_csv(date) 9004 9216 200 1.1 900.4 0.2X write dates to files 6519 6615 148 1.5 651.9 0.2X OpenJDK 64-Bit Server VM 11.0.10+9-LTS on Linux 5.4.0-1043-azure Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz Read dates and timestamps: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------ read timestamp text from files 2364 2428 90 4.2 236.4 1.0X read timestamps from files 39799 39885 96 0.3 3979.9 0.1X infer timestamps from files 79083 79866 818 0.1 7908.3 0.0X read date text from files 2174 2202 37 4.6 217.4 1.1X read date from files 17997 18249 218 0.6 1799.7 0.1X infer date from files 21635 21893 223 0.5 2163.5 0.1X timestamp strings 2686 2719 28 3.7 268.6 0.9X parse timestamps from Dataset[String] 40089 41277 1071 0.2 4008.9 0.1X infer timestamps from Dataset[String] 78144 78581 524 0.1 7814.4 0.0X date strings 2899 2974 74 3.4 289.9 0.8X parse dates from Dataset[String] 19762 19875 99 0.5 1976.2 0.1X from_csv(timestamp) 39890 40339 725 0.3 3989.0 0.1X from_csv(date) 18501 18740 214 0.5 1850.1 0.1X OpenJDK 64-Bit Server VM 11.0.10+9-LTS on Linux 5.4.0-1043-azure Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz Filters pushdown: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------ w/o filters 25642 25863 202 0.0 256422.1 1.0X pushdown disabled 24703 25195 476 0.0 247029.0 1.0X w/ filters 1184 1209 27 0.1 11842.4 21.7X