HFileOutputFormat與TotalOrderPartitioner

最近需要為一些數據增加隨機讀的功能，于是采用生成HFile再bulk load進HBase的方式。

運行的時候map很快完成，reduce在sort階段花費時間很長，reducer用的是KeyValueSortReducer而且只有一個，這就形成了單reducer全排序的瓶頸。于是就想著采用TotalOrderPartitioner使得MR Job可以有多個reducer，來提高并行度解決這個瓶頸。

于是動手寫代碼，不僅用了TotalOrderPartitioner，還使用InputSampler.RandomSampler生成分區文件。但執行時碰到問題，查資料時無意發現HFileOutputFormat內部是使用TotalOrderPartitioner來進行全排序的，

 public static void configureIncrementalLoad(Job job, HTable table)throws IOException {Configuration conf = job.getConfiguration();Class<? extends Partitioner> topClass;try {topClass = getTotalOrderPartitionerClass();} catch (ClassNotFoundException e) {throw new IOException("Failed getting TotalOrderPartitioner", e);}job.setPartitionerClass(topClass);......

?

分區文件的內容就是各region的startKey(去掉最小的)，

private static void writePartitions(Configuration conf, Path partitionsPath,List<ImmutableBytesWritable> startKeys) throws IOException {if (startKeys.isEmpty()) {throw new IllegalArgumentException("No regions passed");}// We're generating a list of split points, and we don't ever// have keys < the first region (which has an empty start key)// so we need to remove it. Otherwise we would end up with an// empty reducer with index 0//沒有哪個rowkey會排在最小的startKey之前，所以去掉最小的startKeyTreeSet<ImmutableBytesWritable> sorted =new TreeSet<ImmutableBytesWritable>(startKeys);ImmutableBytesWritable first = sorted.first();//如果最小的region startKey不是“法定”的最小rowkey，那就報異常if (!first.equals(HConstants.EMPTY_BYTE_ARRAY)) {throw new IllegalArgumentException("First region of table should have empty start key. Instead has: "+ Bytes.toStringBinary(first.get()));}sorted.remove(first);// Write the actual fileFileSystem fs = partitionsPath.getFileSystem(conf);SequenceFile.Writer writer = SequenceFile.createWriter(fs,conf, partitionsPath, ImmutableBytesWritable.class, NullWritable.class);try {//寫入分區文件中 for (ImmutableBytesWritable startKey : sorted) {writer.append(startKey, NullWritable.get());}} finally {writer.close();}}

因為我的表都是新表，只有一個region， 所以肯定是只有一個reducer了。

既然如此，使用HFileOutputFormat時reducer的數量就是HTable的region數量，如果使用bluk load HFile的方式導入巨量數據，最好的辦法是在定義htable是就預先定義好各region。這種方式其實叫Pre-Creating Regions，PCR還能帶來些別的優化，比如減少split region的操作：淘寶有些優化就是應用PCR并且關閉自動split，等到系統空閑時再手動split，這樣可以保證系統繁忙時不會再被split雪上加霜。

關于Pre-Creating Regions: http://hbase.apache.org/book.html#precreate.regions

?11.7.2. Table Creation: Pre-Creating Regions Tables in HBase are initially created with one region by default. For bulk imports, this means that all clients will write to the same region until it is large enough to split and become distributed across the cluster. A useful pattern to speed up the bulk import process is to pre-create empty regions. Be somewhat conservative in this, because too-many regions can actually degrade performance. There are two different approaches to pre-creating splits. The first approach is to rely on the default HBaseAdmin strategy (which is implemented in Bytes.split)...

byte[] startKey = ...;       // your lowest keuy
byte[] endKey = ...;           // your highest key
int numberOfRegions = ...;    // # of regions to create
admin.createTable(table, startKey, endKey, numberOfRegions);

And the other approach is to define the splits yourself...

byte[][] splits = ...;   // create your own splits
admin.createTable(table, splits);

?

?

?