Starting today, the AWS Command Line Interface (AWS CLI) and Python SDK automatically use the AWS Common Runtime (CRT) to accelerate data transfer between Amazon S3 and Amazon EC2 Trn1, P4d, and P5 instances. The AWS CRT implements Amazon S3’s performance best practices for request parallelization, automatic retry, DNS load balancing, and more to deliver high data transfer rates between Amazon EC2 and Amazon S3. As a result, machine learning training jobs now download training data from Amazon S3 up to 3x faster and upload model checkpoints to Amazon S3 up to 5x faster, which accelerates total training times.
Source:: Amazon AWS