背景介绍

NSF(Netease Service Framework)是网易数帆下的一款微服务框架，项目在压测过程中，发现NSF FeignClient有性能瓶颈，下面对遇到的问题及优化方案进行分析，以备忘。

问题分析

在压测过程中发现系统耗时高，通过arthas thread -b发现大量线程处于阻塞状态，进一步分析发现NSF FeignClient使用的java.net.HttpURLConnection，而NSF FeignClient当前版本没有方便的方式（比如改个配置）来支持其他HttpClient，以下通过对NSF FeignClient实现分析来尝试解决这个问题。

NSF FeignClient实现方式

NSF是通过nsf-agent将自定义的FeignClient注入到Spring容器中的，下面是NSF FeignClient的一些主要实现逻辑。

SpringConfigurationTransformer

该类向Spring注入了一些Configuration组件。

public class SpringConfigurationTransformer implements NoneNamedTransformer {
  private static Logger logger = AgentLogFactory.getLogger(SpringConfigurationTransformer.class);
  
  public TransformerResult transform(ClassLoader paramClassLoader, String paramString, Class<?> paramClass, ProtectionDomain paramProtectionDomain, byte[] paramArrayOfByte, Object paramObject) throws Exception {
    ClassPool classPool = ClassPool.getDefault();
    InputStream ins = new ByteArrayInputStream(paramArrayOfByte);
    CtClass ctclass = classPool.makeClass(ins);
    TransformerResult result = new TransformerResult(ctclass);
    result.setClassModified(Boolean.valueOf(true));
    logger.debug("nsf-agent: SpringConfigurationTransformer process transform ...");
    List<String> importConfigurationList = new ArrayList<>();
    importConfigurationList.add("com.netease.cloud.nsf.agent.http.configuration.SpringContextConfiguration");
    importConfigurationList.add("com.netease.cloud.nsf.agent.http.configuration.NsfRegistryConfiguration");
    importConfigurationList.add("com.netease.cloud.nsf.agent.http.configuration.SpringFeignConfiguration");
    importConfigurationList.add("com.netease.cloud.nsf.agent.http.configuration.SpringRibbonConfiguration");
    importConfigurationList.add("com.netease.cloud.nsf.agent.http.configuration.ServletFilterConfiguration");
    importConfigurationList.add("com.netease.cloud.nsf.agent.http.configuration.ToolsConfiguration");
    importConfigurationList.add("com.netease.cloud.nsf.agent.http.configuration.NacosToolsConfiguration");
    importConfigurationList.add("com.netease.cloud.nsf.agent.core.autoconfig.ConfigDebugConfiguration");
    importConfigurationList.add("com.netease.cloud.nsf.agent.http.configuration.NsfSpringDiscoveryClientConfiguration");
    JavassistUtil.addImportAnnotations(ctclass, importConfigurationList);
    result.setClassCodeBytes(ctclass.toBytecode());
    ctclass.defrost();
    return result;
  }
  
  public Object classNameMatch(String paramString) {
    if (paramString.equalsIgnoreCase(GlobalConfigManager.getSpringBootClassName()))
      return Boolean.valueOf(true); 
    return null;
  }
}
复制

与该问题直接相关的是SpringFeignConfiguration。

SpringFeignConfiguration

将自定义的feignClient相关配置注入Spring。

@Configuration
@ConditionalOnClass(name = {"feign.Client", "org.springframework.cloud.openfeign.FeignClient"})
public class SpringFeignConfiguration {
  private Logger logger = AgentLogFactory.getLogger(SpringFeignConfiguration.class);
  @PostConstruct
  public void initFeign() {
    this.logger.info("find feign, init NsfFeignClient ...");
  }
  @Bean(name = {"feignClient"})
  @ConditionalOnClass(name = {"org.springframework.cloud.openfeign.ribbon.LoadBalancerFeignClient"})
  public Client loadBalancerFeignClient(RestTemplate nsfMethodRestTemplate) {
    return (Client)new NsfLoadBalancerFeignClient(nsfMethodRestTemplate);
  }
  
  @Bean(name = {"feignClient"})
  @ConditionalOnMissingClass({"org.springframework.cloud.openfeign.ribbon.LoadBalancerFeignClient"})
  public NsfFeignClient feignClient(RestTemplate nsfMethodRestTemplate) {
    return new NsfFeignClient(nsfMethodRestTemplate);
  }
}
复制

在运行环境中，加载的是NsfLoadBalancerFeignClient。

NsfLoadBalancerFeignClient

public class NsfLoadBalancerFeignClient extends LoadBalancerFeignClient implements Client {
  private NsfFeignClient feignClient;
  public NsfLoadBalancerFeignClient(RestTemplate restTemplate) {
    super((Client)new Client.Default(null, null), null, null);
    this.feignClient = new NsfFeignClient(restTemplate);
  }
  public Response execute(Request request, Request.Options options) throws IOException {
    return this.feignClient.execute(request, options);
  }
}
复制

我们接着查看NsfFeignClient的实现。

NsfFeignClient

public class NsfFeignClient implements Client {
  private static final String ACCEPT_HEADER_NAME = "Accept";
  private ClientHttpRequestFactory requestFactory;
  public NsfFeignClient(RestTemplate restTemplate) {
    this.requestFactory = (ClientHttpRequestFactory)new InterceptingClientHttpRequestFactory((ClientHttpRequestFactory)new NsfClientHttpRequestFactory(), restTemplate.getInterceptors());
  }
  ... ...
}
复制

其中requestFactory是通过NsfClientHttpRequestFactory进行初始化的，也是问题的关键所在，下面看下NsfClientHttpRequestFactory的实现。

NsfClientHttpRequestFactory

public class NsfClientHttpRequestFactory extends SimpleClientHttpRequestFactory {
  public static final ThreadLocal<TimeOut> timeoutCache = new ThreadLocal<TimeOut>() {
      protected NsfClientHttpRequestFactory.TimeOut initialValue() {
        return new NsfClientHttpRequestFactory.TimeOut();
      }
    };
  ... ...
}
复制

NsfClientHttpRequestFactory继承了Spring中的SimpleClientHttpRequestFactory，下面看下SimpleClientHttpRequestFactory中创建连接的逻辑。

SimpleClientHttpRequestFactory

@Override
public ClientHttpRequest createRequest(URI uri, HttpMethod httpMethod) throws IOException {
	HttpURLConnection connection = openConnection(uri.toURL(), this.proxy);
	prepareConnection(connection, httpMethod.name());

	if (this.bufferRequestBody) {
		return new SimpleBufferingClientHttpRequest(connection, this.outputStreaming);
	}
	else {
		return new SimpleStreamingClientHttpRequest(connection, this.chunkSize, this.outputStreaming);
	}
}
protected HttpURLConnection openConnection(URL url, @Nullable Proxy proxy) throws IOException {
	URLConnection urlConnection = (proxy != null ? url.openConnection(proxy) : url.openConnection());
	if (!(urlConnection instanceof HttpURLConnection)) {
		throw new IllegalStateException(
			"HttpURLConnection required for [" + url + "] but got: " + urlConnection);
	}
	return (HttpURLConnection) urlConnection;
}
复制

其中创建连接的逻辑在openConnection方法中，使用HttpURLConnection创建的连接。

优化方案

通过问题分析知道了问题所在，优化方案就比较简单了，只要想办法替换掉java.net.HttpURLConnection就可以了，为了尽可能小的影响NSF FeignClient已有逻辑，我们仿照NSF定义一套相应的类，只需要替换掉java.net.HttpURLConnection的逻辑就可以了。

自定义FeignClient

仿照现有NSF FeignClient单独写一套即可，关键是替换掉java.net.HttpURLConnection。

替换NSF feignClient

通过BeanDefinitionRegistry使用我们自定义的feignClient替换掉NSF feignClient。

public class DefaultBeanDefinitionRegistryPostProcessor implements BeanDefinitionRegistryPostProcessor {
    @Override
    public void postProcessBeanDefinitionRegistry(BeanDefinitionRegistry beanDefinitionRegistry) throws BeansException {
        String[] beanDefinitionNames = beanDefinitionRegistry.getBeanDefinitionNames();
        for(String beanName : beanDefinitionNames){
            if(beanName.equals("feignClient")){
                BeanDefinition wlFeignClient = beanDefinitionRegistry.getBeanDefinition("wlFeignClient");
                if(wlFeignClient != null) {
                    beanDefinitionRegistry.removeBeanDefinition("feignClient");
                    beanDefinitionRegistry.removeBeanDefinition("wlFeignClient");
                    beanDefinitionRegistry.registerBeanDefinition("feignClient", wlFeignClient);
                }
                break;
            }
        }
    }

    @Override
    public void postProcessBeanFactory(ConfigurableListableBeanFactory configurableListableBeanFactory) throws BeansException {
    }
}
复制

总结

通过arthas定位出问题原因，通过分析搞清楚NSF FeignClient的实现，然后再实现一套支持Apache HttpClient的方案来替换原来的HttpURLConnection，经过测试NSF FeignClient已经不再成为阻塞点了。