Build a Data Lake using Lake Formation on AWS

What is a Data Lake?

A Data Lake is a centralized repository for large amounts of structured and unstructured data to enable direct analytics.

Data Lakes provides the ability to harness more data from multiple sources in less time. They empower users to collaborate and analyse data in different ways, which leads to better, faster decisions.

Challenges in building Data Lakes?

Building a data lake is a time-consuming process and there are multiple challenges that you need to address before creating a data lake. The main challenge with a data lake architecture is that raw data is stored without oversight of the contents. For a data lake to make data usable, it needs to have defined mechanisms to catalog and secure data. Without these elements, data cannot be found or trusted, resulting in a “data swamp”. Data lakes require governance, semantic consistency, and access controls to meet the needs of wider audiences. Lets explore these challenges with the below visual representation

Introducing AWS Lake Formation

Lake Formation is a fully managed service provided by AWS that enables data engineers, security officers, and data analysts to build, secure, manage, and use data lake.

There are 3 stages to creating your data lake

Stage 1 – Register data lake storage locations

Stage 2 – Create a database in the data lake’s Data Catalog

Stage 3 – Grant permissions to data lake resources and underlying data

We will demo building data lake for COVID-19 data using AWS Lake Formation and walkthrough all the steps involved in building, securing, managing, and using the same. We will setup the following in this post,

1. Setup a Data Location with sample files on S3.
2. Setup AWS Lake Formation and Create a database
3. Setup Glue Crawler to collect table metadata
4. Query data using Athena

1. Setup a Raw Data Location in a S3 Bucket

We are going to create a sample bucket to store raw data for COVID-19. Bing COVID-19 data includes confirmed, fatal, and recovered cases from all regions, updated daily. This data is reflected in the Bing COVID-19 Tracker.

The data set can be downloaded here. Following is the list of columns and its sample values.

Copy the file to S3 using AWSCLI

aws s3 cp bing_covid-19_data.parquet s3://[BUCKET_NAME]/source

Note: I am using s3://sa-proj-datalake bucket and my IAM user sandeep.arora in my Personal AWS account for this entire walkthrough. Please update the placeholders if you intend to recreate the steps in your environment. The user and bucket has been removed after recording the steps.

2. Setup AWS Lake Formation

To setup data lake we will do the following,

1. Define one or more administrators who will have full access to the lake formation system and they will be responsible for controlling initial data configuration and access permissions
2. Register the S3 path
3. Create a database
4. Provide necessary permissions for the users to access the data lake

2.1 Define Administrators of Data Lake Formation System

1. Navigate to the AWS Lake Formation and under the Permissions section choose Administrative roles and tasks

2. Under the Data lake administrators section add your user that you are logged in with and select additional IAM Users or Roles that you want to promote to lake formation administrators

I have added my IAM user as Administrator to Lake Formation system as shown below,

3. In the Data Creators section ensure that IAMAllowedPrincipals Group is granted Create database permissions

Now that we have successfully setup Administrators for the lake formation system we will proceed with registering S3 location.

2.2 Register S3 Location

Now we will register the location on S3 where we have our raw data set stored

1. Navigate to Data Lake Locations and then from Register and ingest section in the lake formation console

2. Choose Register location to include the S3 storage location as a part of the data lake

3. Register the data set source location in Lake formation and use the service linked role. You must have permissions to create\modify IAM roles to use the service linked role

4. Navigate to IAM console, search for the IAM role and view its attached policies

5. When we registered the location in Lake formation it automatically created the following inline policy and attached it to service linked role and this inline policy is managed by Lake Formation

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "LakeFormationDataAccessPermissionsForS3",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::sa-proj-datalake/source/*"
            ]
        },
        {
            "Sid": "LakeFormationDataAccessPermissionsForS3ListBucket",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::sa-proj-datalake"
            ]
        }
    ]
}

6. Ensure that the location has been registered successfully as shown below,

2.3 Grant access to data location and validate permissions

1. Navigate to Data location > Permissions section

2. Lake formation allows you to manage access for users, roles, external accounts and etc. We are going to grant permissions to data location to current user

3. Grant your current user that you are logged in with permissions to data location as shown below,

4. Validate permissions for databases and tables. Navigate to Data Catalog > Settings and check if IAM access control is enabled for databases and tables

2.4 Create a Database

1. To generate metadata and store them into a data catalog we need to create a database. To create a database using the Lake Formation console, you must be signed in as a data lake administrator or database creator. Navigate to Data catalog > Databases and then create a database

2. Provide the source S3 location and database name as shown below and create the database

3. Setup crawler to determine schema of the table using AWS Glue

We will setup a crawler in AWS Glue to connect to data store, determine the schema and create metadata table(s) in data catalog.

1. Navigate to AWS Glue and from the navigation pane choose Tables from Data Catalog section

2. Add tables using a crawler and this will create a metadata tables in data catalog

3. Provide a Crawler Name

4. Specify the Source type as data stores

5. Specify the data source path to crawl data in

6. Create a new service linked role with necessary permissions to crawl the data source

7. Define Frequency for crawler to run

8. Select the database for crawler output. This is the same database created in Lake formation

9. Review and create the crawler

10. Run the crawler

11. Wait for Tables added column (as shown below) to change to 1 and that would signal the completion of crawler task and table schema being updated in the catalog. Any tables updated or added by Crawler in the catalog are represented by these columns in the console.

12. Navigate back to Lake formation and Data Catalog > Tables. The table should be been added by crawler and its schema must also be populated

Note: The name of the table is selected by the catalog table naming algorithm and you can’t change it later in the console. You can manually add tables if you need to manage names for tables in the console. To do this, when you define a crawler, instead of specifying one or more data stores as the source of a crawl, you specify one or more existing Data Catalog tables. The crawler then crawls the data stores specified by the catalog tables. In this case, no new tables are created; instead, your manually created tables are updated.

4. Query the data using Athena

1. Navigate to AWS Athena and go settings and update the query result location path for S3 as shown below,

2. Select the database and run the query view results of the table

Manage Permissions to data

Database Level Permissions: You can grant or revoke permissions to IAM users or roles to databases in Lake formation using the console

New users, roles, external users or active directory users can be added and database permissions can be granted to level of access needed to the database. See below image for reference.

Table Level Permissions: You can grant or revoke permissions to IAM users to tables using the console. Table level permissions can be granular level and you can define select\insert\delete access to the table. See below image for reference.

Summary

We have successfully setup a data lake, database and data catalog using Lake formation. Lake formation also allows us manage access to data lake resources at database and at table level which helps manage permissions for data lake resources. Overall, Lake formation simplifies the process of setting up data lakes in AWS. I hope this was informative and I would like to thank you for reading.