当前位置：首页 > news >正文

StarRocks StreamLoad 持续写入导致 be 内存增长

news 2026/6/30 23:05:58

背景：Flink 消费 Kafka 通过 StreamLoad 持续写入 StarRocks，be 节点内存持续增长直至到达 be进程内存资源上限，任务中断

查看 be 内存资源

[root@jys-cce-nodepool-bigdata-qnkr7 curl]# kubectl --kubeconfig /opt/module/jys-uat.yaml --namespace bigdata exec -it curl-d6b588446-67h6d -- curl -XGET -s http://kube-starrocks-be-service.starrocks.svc.cluster.local:8040/metrics | grep "^starrocks_be_.*_mem_bytes"starrocks_be_bitmap_index_mem_bytes2772800starrocks_be_bloom_filter_index_mem_bytes2378696starrocks_be_clone_mem_bytes0### 内存占用大头starrocks_be_column_metadata_mem_bytes310631160starrocks_be_column_zonemap_index_mem_bytes108817696starrocks_be_compaction_mem_bytes0starrocks_be_connector_scan_pool_mem_bytes0starrocks_be_consistency_mem_bytes0starrocks_be_datacache_mem_bytes78998594starrocks_be_jit_cache_mem_bytes0starrocks_be_load_mem_bytes0### metadata 内存占用大头starrocks_be_metadata_mem_bytes525319928starrocks_be_ordinal_index_mem_bytes60566288### be 进程starrocks_be_process_mem_bytes1366550648starrocks_be_query_mem_bytes0starrocks_be_rowset_metadata_mem_bytes25569895starrocks_be_schema_change_mem_bytes0starrocks_be_segment_metadata_mem_bytes72609652starrocks_be_segment_zonemap_mem_bytes63078656starrocks_be_short_key_index_mem_bytes2230starrocks_be_storage_page_cache_mem_bytes78998594starrocks_be_tablet_metadata_mem_bytes116509221starrocks_be_tablet_schema_mem_bytes37557starrocks_be_update_mem_bytes0

select*frominformation_schema.be_metricswherenamelike'%_mem_bytes%';

be 内存资源不足

W20260628 02:45:59.098022139791567357504load_channel.cpp:259]tablet writeraddchunk,id=a941cc85-a8fd-86e9-7ac2-4f88d21dae8d,index_id=93207,sender_id=0request_index=0eos=0err=kube-starrocks-be-0.kube-starrocks-be-search.starrocks.svc.cluster.local: memory limit exceeded, please reduce load frequency or increase config`load_process_max_memory_hard_limit_ratio`oraddmoreBE nodes W20260628 02:45:59.098243139793941890624tablet_sink_sender.cpp:126]NodeChannel[10001], tabletaddchunk failed,load_id=a941cc85-a8fd-86e9-7ac2-4f88d21dae8d, txn_id:86459,parallel=1,compress_type=2,node=172.17.251.5:8060,errmsg=kube-starrocks-be-0.kube-starrocks-be-search.starrocks.svc.cluster.local: memory limit exceeded, please reduce load frequency or increase config`load_process_max_memory_hard_limit_ratio`oraddmoreBE nodes

StarRocks 内存变化

sink.batch.size=5000通过以下内存变化，较小的批次现象是：metadata_mem_bytes，column_metadata_mem_bytes占内存资源大头

sink.batch.size=100000通过以下内存变化，10万批次现象是：update_mem_bytes占内存资源大头

参考：

官网指标

column_metadata 内存居高不下

衍生问题

be 统计 tabletNum 数与 statistic 统计不一致

be TabletNum: 25498

mysql>showbackends;+-----------+--------------------------------------------------------------------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------+----------------+--------------------------------------------------------+-------------------+-------------+----------+----------+-------------------+------------+------------+----------------------------------------------------------+----------+------------+|BackendId|IP|HeartbeatPort|BePort|HttpPort|BrpcPort|LastStartTime|LastHeartbeat|Alive|SystemDecommissioned|ClusterDecommissioned|TabletNum|DataUsedCapacity|AvailCapacity|TotalCapacity|UsedPct|MaxDiskUsedPct|ErrMsg|Version|Status|DataTotalCapacity|DataUsedPct|CpuCores|MemLimit|NumRunningQueries|MemUsedPct|CpuUsedPct|DataCacheMetrics|Location|StatusCode|+-----------+--------------------------------------------------------------------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------+----------------+--------------------------------------------------------+-------------------+-------------+----------+----------+-------------------+------------+------------+----------------------------------------------------------+----------+------------+|10001|kube-starrocks-be-0.kube-starrocks-be-search.starrocks.svc.cluster.local|9050|9060|8040|8060|2026-06-2813:15:05|2026-06-2816:48:51|true|false|false|25498|1.183GB|875.371GB|1006.850GB|13.06%|13.06%||4.0.11-9559176|{"lastSuccessReportTabletsTime":"2026-06-28 16:48:51"}|876.554GB|0.14%|8|6.480GB|0|22.48%|0.0%|Status: Normal,DiskUsage:0B/660GB,MemUsage:2MB/1.2GB||OK|+-----------+--------------------------------------------------------------------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------+----------------+--------------------------------------------------------+-------------------+-------------+----------+----------+-------------------+------------+------------+----------------------------------------------------------+----------+------------+1rowinset(0.00sec)

fe 记录 170 tabletNum

mysql>SHOWPROC'/statistic';+-------+--------------------+----------+--------------+----------+-----------+------------+--------------------+-----------------------+------------------+---------------------+|DbId|DbName|TableNum|PartitionNum|IndexNum|TabletNum|ReplicaNum|UnhealthyTabletNum|InconsistentTabletNum|CloningTabletNum|ErrorStateTabletNum|+-------+--------------------+----------+--------------+----------+-----------+------------+--------------------+-----------------------+------------------+---------------------+|10002|_statistics_|12|33|33|170|170|0|0|0|0||1|information_schema|0|0|0|0|0|0|0|0|0||92843|ods|0|0|0|0|0|0|0|0|0||100|sys|0|0|0|0|0|0|0|0|0||Total|4|12|33|33|170|170|0|0|0|0|+-------+--------------------+----------+--------------+----------+-----------+------------+--------------------+-----------------------+------------------+---------------------+5rowsinset(0.00sec)

mysql>SHOWPROC'/backends/10001';+---------------------------+------------------+-------------------+---------------+---------------+--------------+--------+----------------------+---------------+-----------+-------------------+-------------+|RootPath|DataUsedCapacity|OtherUsedCapacity|AvailCapacity|TotalCapacity|TotalUsedPct|State|PathHash|StorageMedium|TabletNum|DataTotalCapacity|DataUsedPct|+---------------------------+------------------+-------------------+---------------+---------------+--------------+--------+----------------------+---------------+-----------+-------------------+-------------+|/opt/starrocks/be/storage|1.183GB|130.260GB|875.407GB|1006.850GB|13.05%|ONLINE|-1197932058022076895|HDD|25498|876.590GB|0.13%|+---------------------------+------------------+-------------------+---------------+---------------+--------------+--------+----------------------+---------------+-----------+-------------------+-------------+1rowinset(0.01sec)

查看 StarRocks 相关配置

查看当前 FE 配置参数

mysql>ADMINSHOWFRONTEND CONFIGLIKE'%trash%';+-----------------------------+------------+-------+------+-----------+---------+|Key|AliasNames|Value|Type|IsMutable|Comment|+-----------------------------+------------+-------+------+-----------+---------+|catalog_trash_expire_second|[]|86400|long|true||+-----------------------------+------------+-------+------+-----------+---------+1rowinset(0.00sec)

be 指标

mysql>select*frominformation_schema.be_metricswherenamelike'%_mem_bytes%';+-------+--------------------------------+--------+------------+|BE_ID|NAME|LABELS|VALUE|+-------+--------------------------------+--------+------------+|10001|bitmap_index_mem_bytes||599200||10001|bloom_filter_index_mem_bytes||444512||10001|chunk_allocator_mem_bytes||0||10001|clone_mem_bytes||0||10001|column_metadata_mem_bytes||75812784||10001|column_zonemap_index_mem_bytes||23616768||10001|compaction_mem_bytes||0||10001|consistency_mem_bytes||0||10001|datacache_mem_bytes||0||10001|jit_cache_mem_bytes||0||10001|load_mem_bytes||0||10001|metadata_mem_bytes||237572764||10001|ordinal_index_mem_bytes||15898064||10001|process_mem_bytes||4519480168||10001|query_mem_bytes||0||10001|rowset_metadata_mem_bytes||132016490||10001|schema_change_mem_bytes||0||10001|segment_metadata_mem_bytes||16589775||10001|segment_zonemap_mem_bytes||13885863||10001|short_key_index_mem_bytes||0||10001|storage_page_cache_mem_bytes||319538||10001|tablet_metadata_mem_bytes||13153715||10001|tablet_schema_mem_bytes||30283||10001|update_mem_bytes||3502819939|+-------+--------------------------------+--------+------------+72rowsinset(0.15sec)

StreamLoad 导入相关

### a.json{"trend_side":"BUY","ask_user_id":14006,"buy_type":1,"sell_fee_coin":"USDT","bid_id":0,"mtime":"2026-02-09 13:07:08","ask_id":0,"volume":0.04,"buy_fee_coin":"DOT","buy_fee":0,"sell_type":1,"price":1.297,"ctime":"2026-02-09 13:07:08","id":23288,"bid_user_id":14006,"sell_fee":0,"source_table":"ex_trade_dotusdt","binlogFile":"snapshot","binlogPosition":0,"op":"r","ts":"2026-06-22 04:50:20","dt":"2026-02-09"}### FE 负载均衡curl--location-trusted\-uroot:123456\-H"label: json_load_002"\-H"format: json"\-H"strip_outer_array: false"\-H"strip_outer_array: false"\-H"Expect: 100-continue"\-Ta.json\http://hadoop02:8030/api/test/ods_ex_trade_history/_stream_load### BE 绕过FEcurl--location-trusted\-uroot:123456\-H"label: json_load_003"\-H"format: json"\-H"strip_outer_array: false"\-H"strip_outer_array: false"\-H"Expect: 100-continue"\-Ta.json\http://hadoop03:8040/api/test/ods_ex_trade_history/_stream_load