Analyze Big Data Platforms For Security and Performance

Development in Intellij

Apache Eagle (called Eagle in the following) can be developed in popular IDE, e.g. Intellij and Eclipse. Here we focus on development in Intellij.

1. Prepare Hadoop1 environment

Normally HDP sandbox is needed for testing Hadoop monitoring. Please reference Quick Start for setting up HDP sandbox.

Please ensure following daemons are up

  • Namenode

  • Datanode

  • HBase2 master/region server only needed when HBase is metadata store

  • MySQL only neede when MySQL is metadata store

  • Zookeeper3

  • Kafka4

2. Start Eagle web service in Intellij

Import source code into Intellij, and find eagle-webservice project. Intellij Ultimate supports launching J2EE server within Intellij. If you don’t have Intellij Ultimate version, Eclipse is another option.

  • Check service configuration

Under eagle-webservice/src/main/resources, configure application.conf for metadata store. As of Eagle 0.4.0, by default, it points to sandbox HBase. If you want to use MySQL as metadata store, just copy application-mysql.conf to application.conf.

  • Ensure tables are created

Run eagle-service-init.sh under eagle-assembly/src/main/bin/eagle-service-init.sh to create metadata tables.

  • Start Eagle service

Configure Intellij for running Apache Tomcat server with eagle-service artifacts

3. Start topology in Intellij

  • Check topology configuration

Take HDFS data activity monitoring as example, config application.conf under eagle-security/eagle-security-hdfs-securitylog/src/main/resources for eagle service endpoint, Kafka endpoint etc.

  • Start Storm5 topology

Normally one data source has one correponding Storm topology, for example to monitor HDFS data activity, org.apache.eagle.security.auditlog.HdfsAuditLogProcessorMain is used for starting Storm topology and reading audit logs from Kafka.

Other example data integrations are as follows

Hive query activity monitoring : org.apache.eagle.security.hive.jobrunning.HiveJobRunningMonitoringMain

HBase data activity monitoring: org.apache.eagle.security.hbaseHbaseAuditLogMonitoringTopology

MapR FS data activity monitoring: org.apache.eagle.security.auditlog.MapRFSAuditLogProcessorMain


Footnotes

  1. All mentions of “hadoop” on this page represent Apache Hadoop.

  2. All mentions of “hbase” on this page represent Apache HBase.

  3. Apache ZooKeeper.

  4. All mentions of “kafka” on this page represent Apache Kafka.

  5. All mentions of “storm” on this page represent Apache Storm.

Copyright © 2015 The Apache Software Foundation, Licensed under the Apache License, Version 2.0.
Apache Eagle, Eagle, Apache Hadoop, Hadoop, Apache HBase, HBase, Apache Hive, Hive, Apache Ambari, Ambari, Apache Spark, Spark, Apache Kafka, Kafka, Apache Storm, Storm, Apache Maven, Maven, Apache Tomcat, Tomcat, Apache Derby, Derby, Apache Cassandra, Cassandra, Apache ZooKeeper, ZooKeeper, Apache, the Apache feather logo, and the Apache project logo are trademarks of The Apache Software Foundation.