Design: New Feature, Status: In Development

Tenets (unless you know better ones)

1. Meeting customers in their problem space allows them to deliver value quickly.
2. Meeting customer expectations drives usability.
3. Discoverability drives usage.
4. Providing a Java-focused experience for DynamoDB reduces the coding effort required to integrate with DynamoDB.
5. Reusing the same nouns and verbs as the generated DynamoDB client meets customer expectations.
6. Optimizing for cold-start performance allows customers the convenience of using object mapping in a Lambda environment.

Problem

Customers on the AWS SDK for Java 2.x currently use the DynamoDbClient to communicate with DynamoDB. This client is generated from the model provided by the DynamoDB team.

Because this client is generated, it does not provide an idiomatic Java experience. For example: (1) the client represents numbers as String instead of the more idiomatic Number, (2) customers must manually convert common Java data types like Instant into types supported by DynamoDB, (3) customers that represent their DynamoDB objects using Java objects must manually convert these objects into the item representation supported by DynamoDB.

Existing Solutions

This problem is not currently addressed directly in the AWS SDK for Java 2.x by any known third-party tool. In 1.11.x, several solutions exist, including AWS's own Document and Mapper Clients.

Proposed Solution

The AWS SDK for Java will add a new “enhanced DynamoDB client” that provides an alternative to the data-access portion of the generated DynamoDB APIs. Control-plane operations like “create table” will not be supported at launch, but may be added at a later time.

This enhanced client will make DynamoDB easier to use for Java customers by:

1. Supporting conversions between Java objects and DynamoDB items
2. Supporting conversions between Java built-in types (eg. Instant) and DynamoDB attribute value types
3. Directly supporting every data-plane operation of DynamoDB
4. Using the same verbs and nouns of DynamoDB

Implementation Overview

New Clients

Two new client classes will be added: DynamoDbEnhancedClient and DynamoDbEnhancedAsyncClient. These classes act as a wrapper around the generated DynamoDbClient and DynamoDbAsyncClient classes, to provide additional functionality on top of that which can be provided by the generated clients.

DynamoDbEnhancedClient enhancedClient = 
    DynamoDbEnhancedClient.builder()
                          .dynamoDbClient(DynamoDbClient.create())
                          .build();

Table Abstraction

DynamoDbEnhancedClient provides access to Table and MappedTable, and DynamoDbEnhancedAsyncClientprovides access to AsyncTable, and AsyncMappedTable abstractions.

The operations on these “tables” match the data-plane operations in the low-level DynamoDB client. For example, because DynamoDbClient.putItem exists, Table.putItem will also exist.

Table and AsyncTable work with “items”, described below. MappedTable and AsyncMappedTable work with “objects”, described below. Table and MappedTable returning results synchronously, and AsyncTable and AsyncMappedTable returning results asynchronously.

Table booksTable = enhancedClient.table("books");
booksTable.putItem(...);

MappedTable mappedBooksTable = enhancedClient.mappedTable("books");
mappedBooksTable.putItem(...);

Item Abstraction

The operations on Table and AsyncTable work on Items. An Item is a user-friendly representation of the generated Map<String, AttributeValue>. Items support automatic type conversion between Java built-in types and DynamoDB-specific AttributeValue types.

booksTable.putItem(Item.builder()
                       .putAttribute("isbn", "0-330-25864-8")
                       .putAttribute("title", "The Hitchhiker's Guide to the Galaxy")
                       .putAttribute("creationDate", Instant.now())
                       .build());

The Table and AsyncTable abstractions can be seen as a replacement for the 1.11.x DynamoDB Document client.

Object Abstraction

The operations on MappedTable and AsyncMappedTable work on Java objects (at launch, Java beans). These objects are automatically converted by the enhanced client to the generated Map<String, AttributeValue>. It's likely that the MappedTable and AsyncMappedTable will use the Table and AsyncTable as an implementation detail.

Book book = new Book();
book.setIsbn("0-330-25864-8");
book.setTitle("The Hitchhiker's Guide to the Galaxy");
book.setCreationDate(Instant.now());
mappedBooksTable.putItem(book);

The MappedTable and AsyncMappedTable abstractions can be seen as a replacement for the 1.11.x DynamoDB Mapper client.

Type Conversion

The core feature of the mapper is the ability to convert common Java structures (e.g. Java beans) and types (e.g. Instant, Number) into DynamoDB attribute values.

These conversions are performed based on the types specified by the customer. For example, the SDK will automatically convert any Number types specified by the customer (as an Item attribute) into a DynamoDB number.

The customer has the ability to configure the type converters used at the Item or DynamoDbEnhanced[Async]Client-level. This allows the customer to add support for unsupported types, change the DynamoDB type associated with a Java type (e.g. storing an Instant as a DynamoDB string instead of a number), or to add support for custom POJO conversion logic (i.e. other than Java beans). This also allows the customer to provide a hard-coded converter for a specific object type that performs more efficiently than the built-in reflection-based object converter.

Features

Launch Features

These features are intended for inclusion at launch of the library.

1. Support for all existing data plane operations: get, put, query, update, scan, delete, batch get, batch put, transaction get, and transaction put.
2. Support for [Async]Table and [Async]MappedTable, as described above.
3. Support for bean-based representations in [Async]MappedTable.
4. Type converters for all Java built-in types that are currently supported by Joda Convert.

Post-Launch Features

1. Support for inheritance in [Async]MappedTable.
2. Support for immutable objects in [Async]MappedTable.
3. Support for projection statements in [Async]Table and [Async]MappedTable.
4. Support for DynamoDB-provided API metrics (e.g. consumed capacity).
5. A software.amazon.aws:dynamodb-all module that automatically includes all AWS DynamoDB artifacts, to enhance client discoverability.

Missing Features

These features are not intended for inclusion at launch of the library (but may be added at a future time).

1. Support for control-plane operations, like create or delete table. Justification for exclusion: For testing purposes, this can be done through the AWS console or low-level SDK. For production purposes, this should be done through the AWS CDK or cloud formation.
2. Versioning and UUID annotations. Justification for exclusion: This is a higher-level concern than the “type converter” goal that the enhanced client is attempting to deliver on. This is a piece of functionality that will be built on-top of the enhanced client, not in it.

Requested Features

• Immutable classes
• Getter/setter-less fields
• Replace PaginatedList with Stream
• Allow ‘setters’ and ‘getters’ to support different types
• Have ‘scan’ respect the table's read throughput
• Allow creating a table with an LSI that projects all attributes
• Projection expressions in ‘load’ and ‘batchLoad’
• New condition expressions
• Accessing un-modeled/dynamic attributes in a POJO
• Inheritance
• Service-side metrics (1, 2, 3)
• Merging DynamoDB mapper configurations
• Cache merged DynamoDB mapper configurations
• Create one single type converter interface
• Support @DynamoDBGeneratedUuid in objects nested within lists
• Allow annotating fields in addition to methods
• Non-string keys in maps
• Multiple conditions on the same attribute, for save/delete
• Persisting public getters from package-private classes
• Return modified attributes when doing a save
• More direct exposure of scan or filter expressions
• Transactions support
• Creating an Item from JSON (and vice-versa)
• Straight-forward support for multiple classes in a single table (as per here) (from email)
• Support for Optional (from email)
• Support for Publisher for async paginated responses (from email)
• Create a table with partial projections (from email)
• Better integration with DynamoDB streams (from email)
• Configuring table auto-scaling when a table is created (from email)
• Request-level credentials (from email)
• Wrappers for transactional isolation (from email)
• Dynamic attributes - ones with different types depending on the value of other attributes, or attributes with names that are generated at runtime (from email)
• Structure versioning (from email)

Appendix A: Alternative Solutions

Alternative Solution 1: Level 3 Storage Library

A “Level 2” high-level library is a service-specific library built on top of the “Level 1” generated client. The solution proposed above is a Level 2 high-level library for DynamoDB.

A “Level 3” high-level library focuses on a specific customer problem instead of a specific AWS service. For example, customers frequently use DynamoDB to store time series data. An alternate to the proposed solution above, would be to build multiple Level 3 libraries, each focusing on a specific customer problem: a document database library, a time series database library, etc. These libraries would support DynamoDB as one of many backing data stores.

Instead of using traditional DynamoDB nouns and verbs (e.g. Item), a Level 3 library would use words more aligned to the problem domain (e.g. Document for document databases or Entry for time-series data). They would also expose operations more constrained to the problem domain they were trying to solve, instead of trying to expose every piece of DynamoDB functionality.

This solution would be better for customers that are more familiar with the problem they are trying to solve and less familiar with DynamoDB. This solution would be worse for customers that are familiar with DynamoDB and want to be “closer” to the service.

Customer Feedback

The Java SDK team collected customer feedback internally and externally, comparing this alternate solution against the proposed solution. Customers were presented with the following option comparison:

Option 1: A DynamoDB-specific client that combines the functionality of 1.11.x's Documents APIs and DynamoDB Mapper APIs in a straight-forward manner.

Option 2: A generic document database client that creates an abstraction over all document databases, like DynamoDB and MongoDB. This would simplify using multiple document databases in the same application, and make it easier to migrate between the two. Unfortunately as a result, it also wouldn't be a direct DynamoDB experience.

We requested that customers review these two options as well as a prototype of option 1 and a prototype of option 2, to let us know which they prefer.

The following anecdotes are from this customer feedback:

If [Amazon] can make something like https://serverless.com/ or https://onnx.ai/ which free customers from vendor lock-in, that would be a great Think Big & Customer Obsession idea. If [Amazon] cannot, I feel that somebody who is more vendor-neutral can make a better mapper than [Amazon].

Have you thought about contributing to projects which already exist, like Spring Data? https://github.com/derjust/spring-data-dynamodb

Both options would work well for us.

I think [doing option 1 and then creating a Spring Data plugin] might get adoption from a broader audience than option 2. It could be used as a stepping stone to move to DynamoDB.

I believe Option 2 does not make much sense. It would make sense to me to go for Option 1 and start a bounty program to implement a module to popular data access abstraction libraries such as spring-data mentioned above or GORM.

Maybe you could implement/support JNOSQL spec http://www.jnosql.org/

Decision

Based on customer feedback, it was decided to temporarily reject alternative solution 1, and build the proposed solution. At a later time, the SDK may build a Level 3 abstraction for DynamoDB or integrate with existing Java Level 3 abstractions like Spring Data, Hibernate OGM, and/or JNoSQL. This Level 3 abstraction will possibly leverage the Level 2 solution “under the hood”.

Links

Features - The features intended for inclusion during and after the launch of the enhanced DynamoDB client.

Prototypes

During the design of the project, two prototype interfaces were created to solicit feedback from customers on potential design directions.

• Prototype 1 - A DynamoDB specific API that focuses on making DynamoDB easy to use from Java.
• Prototype 2 - A DynamoDB-agnostic API that focuses on creating a generic document database abstraction, that could be backed by DynamoDB or other document databases.

Feedback

• DynamoDB Mapper Feature Request - A github issue for tracking customer feature requests and feedback for DynamoDB mapper-equivalent functionality in 2.x.
• DynamoDB Document API Feature Request - A github issue for tracking customer feature requests and feedback for DynamoDB document API-equivalent functionality in 2.x.