Support Support Downloads Knowledge Base Case Manager My Juniper Community

Knowledge Base

Search our Knowledge Base sites to find answers to your questions.

Ask All Knowledge Base Sites All Knowledge Base Sites JunosE Defect (KA)Knowledge BaseSecurity AdvisoriesTechnical BulletinsTechnotes Sign in to display secure content and recently viewed articles

[Contrail] Change Cassandra LeveledCompactionStrategy on builds older than 2016-12-08

0

0

Article ID: KB31843 KB Last Updated: 22 Jun 2017Version: 1.0
Summary:

Leveled Compaction Strategy (LCS) causes load increase, especially when running on Hard Disk Drive (HDD). This is the default compacting strategy on builds older than 2016-12-08.

Symptoms:
With the increase of usage, Cassandra is no longer able to keep up with the compaction. As a result, you may be slowly running out of disk space for analytics.
Solution:

Try the method described in Contrail Analytics DB data purge

If that does not help, and you are running a build older than 2016-12-08, change the compacting strategy manually, as you are affected by the performance limitation described in LP1648252.

Use the following procedure to change the compaction strategy for the analytics tables in Cassandra. The procedure uses 2 files:
  • cql-truncate-table-analytics
  • cql-alter-compaction-strategy-analytics
  1. Run service supervisor-analytics stop on all nodes

  2. Truncate analytics table and alter compaction strategy

        a. cqlsh <1st-database-node-ip> -f cql-truncate-table-analytics
        b. nodetool status, should show truncate command has cleared the data - data in MB rather than GB
        c. cqlsh <1st-database-node-ip> -f cql-alter-compaction-strategy-analytics
        d. cqlsh <1st-database-node-ip> to verify that the compaction strategy is changed
           >>> describe keyspace "ContrailAnalyticsCql"
           should show new compaction strategy
Note: Steps 3 and 4 are required only if you are changing concurrent_compactors
  1. Change concurrent_compactors to 4 in /etc/cassandra/cassandra.yaml on all nodes

  2. Rolling restart of cassandra

    On each node serially:
        a. nodetool drain
        b. service contrail-database restart
        c. nodetool status to verify that it has joined the cluster
  3. service supervisor-analytics start on all nodes

Files used in fix

Create the following files to be used in the fix provided above.

cql-truncate-table-analytics:

truncate table "ContrailAnalyticsCql".flowrecordtable;
truncate table "ContrailAnalyticsCql".objectvaluetable;
truncate table "ContrailAnalyticsCql".messagetabletimestamp;
truncate table "ContrailAnalyticsCql".statstablebyu64tagv3;
truncate table "ContrailAnalyticsCql".systemobjecttable;
truncate table "ContrailAnalyticsCql".objecttable;
truncate table "ContrailAnalyticsCql".messagetable;
truncate table "ContrailAnalyticsCql".flowtablesvnsipver2;
truncate table "ContrailAnalyticsCql".flowtablevrouterver2;
truncate table "ContrailAnalyticsCql".messagetablemessagetype;
truncate table "ContrailAnalyticsCql".messagetablecategory;
truncate table "ContrailAnalyticsCql".messagetablesource;
truncate table "ContrailAnalyticsCql".flowtableprotdpver2;
truncate table "ContrailAnalyticsCql".statstablebyu64u64tagv3;
truncate table "ContrailAnalyticsCql".statstablebystrtagv3;
truncate table "ContrailAnalyticsCql".messagetablemoduleid;
truncate table "ContrailAnalyticsCql".statstablebydbltagv3;
truncate table "ContrailAnalyticsCql".messagetablekeyword;
truncate table "ContrailAnalyticsCql".statstablebyu64strtagv3;
truncate table "ContrailAnalyticsCql".statstablebystrstrtagv3;
truncate table "ContrailAnalyticsCql".statstablebystru64tagv3;
truncate table "ContrailAnalyticsCql".flowtabledvndipver2;
truncate table "ContrailAnalyticsCql".flowtableprotspver2;


cql-alter-compaction-strategy-analytics:

alter table "ContrailAnalyticsCql".flowrecordtable WITH compaction = {'class': 'org.apache.cassandra.db.compaction.DateTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".objectvaluetable WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".messagetabletimestamp WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".statstablebyu64tagv3 WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".systemobjecttable WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".objecttable WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".messagetable WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".flowtablesvnsipver2 WITH compaction = {'class': 'org.apache.cassandra.db.compaction.DateTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".flowtablevrouterver2 WITH compaction = {'class': 'org.apache.cassandra.db.compaction.DateTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".messagetablemessagetype WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".messagetablecategory WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".messagetablesource WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".flowtableprotdpver2 WITH compaction = {'class': 'org.apache.cassandra.db.compaction.DateTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".statstablebyu64u64tagv3 WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".statstablebystrtagv3 WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".messagetablemoduleid WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".statstablebydbltagv3 WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".messagetablekeyword WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".statstablebyu64strtagv3 WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".statstablebystrstrtagv3 WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".statstablebystru64tagv3 WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".flowtabledvndipver2 WITH compaction = {'class': 'org.apache.cassandra.db.compaction.DateTieredCompactionStrategy'};
alter table "ContrailAnalyticsCql".flowtableprotspver2 WITH compaction = {'class': 'org.apache.cassandra.db.compaction.DateTieredCompactionStrategy'};

Finishing step & cleanup

According to the Cassandra documentation, you should set read repair to 0 when using DateTieredCompactionStrategy:

"The compaction strategy DateTieredCompactionStrategy precludes using read repair, because of the way timestamps are checked for DTCS compaction.
In this case, you must set read_repair_chance to zero. For other compaction strategies, read repair should be enabled with a read_repair_chance value of 0.2 being typical." So be sure to add the following:

alter table "ContrailAnalyticsCql".flowrecordtable WITH read_repair_chance = 0.0;
alter table "ContrailAnalyticsCql".flowtablesvnsipver2 WITH read_repair_chance = 0.0;
alter table "ContrailAnalyticsCql".flowtablevrouterver2 WITH read_repair_chance = 0.0;
alter table "ContrailAnalyticsCql".flowtableprotdpver2 WITH read_repair_chance = 0.0;
alter table "ContrailAnalyticsCql".flowtabledvndipver2 WITH read_repair_chance = 0.0;
alter table "ContrailAnalyticsCql".flowtableprotspver2 WITH read_repair_chance = 0.0;


Truncate command in the procedure will create snapshot directories. To clean that data, use "nodetool clearsnapshot" on each node.
Comment on this article > Affected Products Browse the Knowledge Base for more articles related to these product categories. Select a category to begin.

Getting Up and Running with Junos

Getting Up and Running with Junos Security Alerts and Vulnerabilities Product Alerts and Software Release Notices Problem Report (PR) Search Tool EOL Notices and Bulletins JTAC User Guide Customer Care User Guide Pathfinder SRX High Availability Configurator SRX VPN Configurator Training Courses and Videos End User Licence Agreement Global Search