今日异常记录20260108 错误信息OutOfMemoryError: unable to create new native thread

今日项目，后台异常报错

java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: unable to create new native thread
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:592)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:551)
at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:111)
at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:247)
at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2345)
at com.google.common.cache.LocalCache$Segment.lambda$loadAsync$0(LocalCache.java:2325)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:760)
at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.addListener(AbstractFuture.java:134)
at com.google.common.cache.LocalCache$Segment.loadAsync(LocalCache.java:2322)
at com.google.common.cache.LocalCache$Segment.refresh(LocalCache.java:2393)
at com.google.common.cache.LocalCache$Segment.scheduleRefresh(LocalCache.java:2370)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2069)
at com.google.common.cache.LocalCache.get(LocalCache.java:4011)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4034)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:5010)
at com.bayi.meta.framework.tenant.core.service.TenantFrameworkServiceImpl.getTenantIds(TenantFrameworkServiceImpl.java:62)
at com.bayi.meta.framework.tenant.core.job.TenantJobHandlerDecorator.execute(TenantJobHandlerDecorator.java:34)
at com.bayi.meta.framework.quartz.core.handler.JobHandlerInvoker.executeInternal(JobHandlerInvoker.java:74)
at com.bayi.meta.framework.quartz.core.handler.JobHandlerInvoker.executeInternal(JobHandlerInvoker.java:57)
at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:75)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:719)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
at com.google.common.cache.CacheLoader$1.reload(CacheLoader.java:199)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3573)
at com.google.common.cache.LocalCache$Segment.loadAsync(LocalCache.java:2321)
	... 13 common frames omitted

docker下的mysql尝试重启，启动成功了一会，然后自己又崩了

2026-01-08T02:06:50.825737Z 0 [ERROR] [MY-010249] [Server] Can't create thread to handle new connection(errno= 11)

问了ai，分析问题，感觉是系统线程资源耗尽，重启容器不行，重启docker也不行，最后只能重启服务器了，

输入reboot之后，大半个小时才启动，还以为启动不了

重启之后目前都正常了，分析后台日志，发现7号下午5点后台开始内存溢出，这之前日志输出是一个socket下发消息的任务，有错误，但是不知道是否关联

java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:256)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:750)

2026-01-07 16:59:52.816 |  WARN 7 | nioEventLoopGroup-10-27 [TID: N/A] c.b.m.m.shelf.netty.TcpServerHandler     | xx导航屏，客户端【/192.168.6.30:2046】断开连接
2026-01-07 16:59:52.836 |  WARN 7 | nioEventLoopGroup-10-28 [TID: N/A] c.b.m.m.shelf.netty.TcpServerHandler     | xxx导航屏，客户端【/192.168.6.30:2037】建立连接
2026-01-07 16:59:52.852 |  INFO 7 | nioEventLoopGroup-10-28 [TID: N/A] c.b.m.m.shelf.api.tcp.ShelfTcpApiImpl    | TCP发送数据，内容：xxx
2026-01-07 16:59:52.852 |  INFO 7 | nioEventLoopGroup-10-28 [TID: N/A] c.b.m.module.shelf.netty.TcpServerUtil   | 已连接客户端：["192.168.6.30"]
2026-01-07 17:00:00.061 |  WARN 7 | schedulerName_Worker-22 [TID: N/A] com.google.common.cache.LocalCache       | Exception thrown during refresh

还是问ai，ai觉得有关系，让我优化soket配置和两处方法的变量清零

@Override
public void channelInactive(ChannelHandlerContext ctx) {
    log.warn("xxx导航屏，客户端【{}】断开连接", ctx.channel().remoteAddress().toString());
    // 连接断开时，主动清空缓冲区，帮助GC
    msgBuffer.setLength(0);
    TcpServerUtil.removeChannel(ctx);
}

/**
 * 异常捕获
 */
@Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
    log.error("xxx导航屏，Netty异常，客户端: {}", ctx.channel().remoteAddress().toString(), cause);
    // 发生异常时，也清空缓冲区
    msgBuffer.setLength(0);
    TcpServerUtil.removeChannel(ctx);
    ctx.close();
}

同时怀疑合作方大批量调用接口也有关系，目前继续观察，如果后续继续有问题，尝试更新代码