Does the solution meet the goal?

Note: This question is part of a series of questions that present the same scenario. Each question in the series contains a unique solution that might meet the stated goals. Some question sets might have more than one correct solution, while others might not have a correct solution.

After you answer a question in this section, you will NOT be able to return to it. As a result, these questions will not appear in the review screen.

You are designing an HDInsight/Hadoop cluster solution that uses Azure Data Lake Gen1 Storage.

The solution requires POSIX permissions and enables diagnostics logging for auditing.

You need to recommend solutions that optimize storage.

Proposed

Solution: Implement compaction jobs to combine small files into larger files.

Does the solution meet the goal?
A . Yes
B . No

View Answer

Answer: A

Explanation:

Depending on what services and workloads are using the data, a good size to consider for files is 256 MB or greater. If the file sizes cannot be batched when landing in Data Lake Storage Gen1, you can have a separate compaction job that combines these files into larger ones.

Note: POSIX permissions and auditing in Data Lake Storage Gen1 comes with an overhead that becomes apparent when working with numerous small files. As a best practice, you must batch your data into larger files versus writing thousands or millions of small files to Data Lake Storage Gen1.

Avoiding small file sizes can have multiple benefits, such as:

– Lowering the authentication checks across multiple files

– Reduced open file connections

– Faster copying/replication

– Fewer files to process when updating Data Lake Storage Gen1 POSIX permissions

Reference: https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-best-practices