Skip to content

NIFI-15717: Add QueryCassandra processor and CassandraSessionProvider Controller Service. - #11032

Closed
vishal-firgan-ksolves wants to merge 5 commits into
apache:mainfrom
vishal-firgan-ksolves:NIFI-15717
Closed

vishal-firgan-ksolves wants to merge 5 commits into
apache:mainfrom
vishal-firgan-ksolves:NIFI-15717

Conversation

@vishal-firgan-ksolves

Copy link
Copy Markdown
Contributor

…r Service.

Summary

NIFI-15717

This PR introduces the QueryCassandra processor for Apache NiFi 2.x, enabling integration with Cassandra using the latest DataStax Java driver.

Components Added

  • QueryCassandra processor for executing CQL queries
  • CassandraSessionProvider Controller Service for managing connections

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

Pull Request Tracking

  • Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
  • Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000
  • Pull request contains commits signed with a registered key indicating Verified status

Pull Request Formatting

  • Pull Request based on current revision of the main branch
  • Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

  • Build completed using ./mvnw clean install -P contrib-check
    • JDK 21
    • JDK 25

Licensing

  • New dependencies are compatible with the Apache License 2.0 according to the License Policy
  • New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

  • Documentation formatting appears as expected in rendered files

@exceptionfactory exceptionfactory left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for putting together this down-scoped pull request. I'm planning to take a closer look soon, but it provides a good basis for initial review.

One initial element to consider is the data type handling in the AbstractCassandraProcessor. It seems like it would be best to pull that out to a separate interface and implementation, perhaps several depending on the details. The general goal there is to decouple type conversion from the Processor itself, so that it can be used in composition, rather than by extension in other Cassandra Processors.

@vishal-firgan-ksolves

Copy link
Copy Markdown
Contributor Author

@exceptionfactory I’ve implemented the changes to decouple type conversion from AbstractCassandraProcessor into a new CassandraTypeConverter interface with StandardCassandraTypeConverter as its implementation.
Please review the changes.

@exceptionfactory exceptionfactory left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the initial update @vishal-firgan-ksolves.

On a more substantive review, I noticed a large number of significant design and implementation issues. I highlighted a number of them, but based on this review, this will require substantial rework before going forward. I'm not in a position to provide direction on every detail, so this may need to be revisited down the road.

Comment thread nifi-assembly/pom.xml Outdated
Comment on lines +750 to +764
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-cassandra-processors</artifactId>
<version>2.9.0-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-cassandra-services</artifactId>
<version>2.9.0-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-cassandra-services-api</artifactId>
<version>2.9.0-SNAPSHOT</version>
</dependency>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Component JAR dependencies do not belong in the nifi-assembly module.

<groupId>org.apache.nifi</groupId>
<artifactId>nifi-cassandra-bundle</artifactId>
<version>2.9.0-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This relative path is not needed and should be removed.

} else if (type.equals(DataTypes.TIME)) {
return Long.class;
} else if (type.equals(DataTypes.BLOB)) {
return java.nio.ByteBuffer.class;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Qualified class name references should be avoided in favor of standard imports.

*/
public abstract class AbstractCassandraProcessor extends AbstractProcessor {

static final PropertyDescriptor CONNECTION_PROVIDER_SERVICE = new PropertyDescriptor.Builder()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Property Descriptor variables should align closely with the property name, such as CASSANDRA_CONNECTION_PROVIDER.

See the License for the specific language governing permissions and
limitations under the License.

APACHE NIFI SUBCOMPONENTS:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This license file is incorrect and contains references that are not applicable.

.build();

static final PropertyDescriptor READ_TIMEOUT_MS = new PropertyDescriptor.Builder()
.name("Read Timeout (ms)")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This convention is not correct, duration validators and syntax should be used.

# See the License for the specific language governing permissions and
# limitations under the License.

org.apache.nifi.service.CassandraSessionProvider No newline at end of file

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be changed to a cassandra subpackage.

/**
* Mock Cassandra processor for testing CassandraSessionProvider
*/
public class MockCassandraProcessor extends AbstractProcessor {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the reason for this class, as opposed to just using Mockito?

Comment thread pom.xml Outdated
Comment on lines +506 to +510
<dependency>
<groupId>org.lz4</groupId>
<artifactId>lz4-java</artifactId>
<version>${lz4-java.version}</version>
</dependency>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This dependency is banned.

@vishal-firgan-ksolves

Copy link
Copy Markdown
Contributor Author

@exceptionfactory ,
Thanks for the detailed review and for pointing out the design and implementation issues.

I understand that the current approach needs significant rework. The initial implementation was based on patterns from NiFi Cassandra processors (around the 1.28 codebase), but I see that these are no longer aligned with current expectations.

I will revisit the implementation and refactor it to better align with current NiFi standards and patterns, and follow up with an updated version.

@vishal-firgan-ksolves

Copy link
Copy Markdown
Contributor Author

@exceptionfactory
I have addressed all review feedback, including refactoring processors and services, and aligning the implementation with current NiFi standards. The implementation has been updated to align with existing NiFi bundle patterns and project structure. Please review the updated changes.

@vishal-firgan-ksolves

Copy link
Copy Markdown
Contributor Author

@exceptionfactory
Following up on the recent updates. The implementation has been refactored to align with current NiFi patterns based on earlier feedback. When you get a chance, could you please take another look.

@exceptionfactory exceptionfactory left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your patience on this pull request @vishal-firgan-ksolves. I have not reviewed all the changes given a number of other issues lately, but I highlighted an additional set of changes needed. This may require more work.

TLS/SSL connections to the Cassandra cluster.
""")
.required(false)
.identifiesControllerService(SSLContextService.class)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should use the more generic SSLContextProvider interface.

Suggested change
.identifiesControllerService(SSLContextService.class)
.identifiesControllerService(SSLContextProvider.class)

Comment on lines +148 to +149
.description("Connection timeout. 0 means no timeout. If no value is set, the underlying default will be used.")
.required(false)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A default value should be provided.

Comment on lines +141 to +142
.description("Read timeout. 0 means no timeout. If no value is set, the underlying default will be used.")
.required(false)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A default value should be provided

Specifies the compression type used for transport-level requests and responses.Note that Snappy
compression is not supported when using Protocol V5.
""")
.required(false)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A property with a default value should be required

public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder()
.name("Username")
.description("Username to access the Cassandra cluster")
.required(false)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the Username be required?

runner.enableControllerService(sessionProvider);

verify(sessionProvider, times(1)).connectToCassandra(any(ConfigurationContext.class));
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test appears to just exercise the spied instance, and should be removed

</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-ssl-context-service-api</artifactId>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be provided

Comment on lines +49 to +53
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-framework-api</artifactId>
<version>2.10.0-SNAPSHOT</version>
</dependency>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be removed

<groupId>at.yawk.lz4</groupId>
<artifactId>lz4-java</artifactId>
<version>1.10.4</version>
</dependency>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be inherited from the parent pom.xml

Comment thread pom.xml
</dependency>
<dependency>
<groupId>org.apache.cassandra</groupId>
<artifactId>java-driver-core</artifactId>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be moved down to the Cassandra bundle

import org.apache.nifi.cassandra.exception.CassandraException;
import org.apache.nifi.cassandra.exception.CassandraExceptionCategory;

public final class DataStaxExceptionMapper {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm thinking for posterity we should change references from Datastax to "CassandraDriver"

@exceptionfactory exceptionfactory left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the work on this PR @vishal-firgan-ksolves. With the current effort to reintroduce Cassandra support in #11595, I'm closing this PR for now. If there are additional features needed, I recommend building on that effort.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants