Get All Rows from DynamoDB Table (Scan or Query) using Node.js

DynamoDB has a maximum limit of 1 MB in its Query or Scan API responses. This means that if requested data is pretty huge in size, we need to make several Query / Scan requests to consume the full data.

• If DynamoDB API's response reaches its maximum limit, it adds an attribute LastEvaluatedKey to the response.
• We need to keep looking for this attribute, and use its value in the ExclusiveStartKey attribute in the next request. Basically LastEvaluatedKey from the last response becomes the ExclusiveStartKey value for the new request.
• If there is no LastEvaluatedKey in the response, this means there are no remaining rows to get.

Install AWS DynamoDB Javascript SDK

npm install @aws-sdk/client-dynamodb

Getting All Rows with Scan

Assume we have a DynamoDB table named Posts. It has the attributes PostID (partition key), PostTitle & PostVisible.

The below code also assumes 3 environment variables have been set — AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY & AWS_REGION.

const { DynamoDBClient, ScanCommand } = require("@aws-sdk/client-dynamodb")

// init dynamodb client
const dbclient = new DynamoDBClient({ region: process.env.AWS_REGION })

// will hold all rows
let allPosts = []

// will hold LastEvaluatedKey attribute
let lastEvaluatedKey

// keep requesting new data until break
while (true) {
  const params = {
    TableName: 'Posts',
	  ProjectionExpression: 'PostID,PostTitle,PostVisible'
  }

  if (lastEvaluatedKey) {
    params.ExclusiveStartKey = lastEvaluatedKey
  }

  const data = await dbclient.send(new ScanCommand(params))

  const reqPosts = []
  data.Items.forEach(post => {
    reqPosts.push(post)
  })
  allPosts = allPosts.concat(reqPosts)

  // if LastEvaluatedKey attribute present & not-null
  // then additional rows are remaining
  if ('LastEvaluatedKey' in data && data.LastEvaluatedKey) {
    lastEvaluatedKey = data.LastEvaluatedKey
  } else {
    break
  }
}

console.log(allPosts)

Getting All Rows with Query

Same logic will hold for Query operations as well (either querying from table or an index). We need to check for LastEvaluatedKey attribute in the response, and set this as the ExclusiveStartKey in the next request.

Read API docs for ScanCommand and QueryCommand.