Envoyproxy introduced its new version, 1.19.0, recently, and it came with many changes and improvements from the previous ones. We can see more stability in this version, along with specific bug fixes. So, without waiting any further, let’s see what the new version has in its store.
Incompatible Behavior Changes
In the new version, we can expect that specific changes will cause incompatibility if applicable, and as a result, we will need some deployment changes. For the
Grpc_bridge_filter, we can see that the filter no longer collects grpc stats favouring the existing grpc stats filter. If you want to revert this specific behaviour, you can change the runtime key
envoy.reloadable_features.grpc_bridge_stats_disabled. Also, we can see tracing the updated Apache SkyWalking tracer version to be compatible with 8.4.0 for the data collecting protocol. This change will introduce an incompatibility with SkyWalking 8.3.0.
Minor Behavior Changes
These are the changes that may cause incompatibilities for some users but should not for most.
The new version again comes with the addition of a new access_log command operator
%REQUEST_TX_DURATION%. We can also see the removal of additional quotes on metadata string values. We can temporarily revert this behaviour by setting
Envoy.reloadable_features.unquote_log_string_values to false.
There is an addition of
max_rejection_probability, which defaults to 80%. It means that the upper limit of the default rejection probability of the filter changes from 100% to 80%.
AWS Request Signing
The new version brings default buffering of requests to compute signatures, including the payload hash, making the filter compatible with most AWS services. Previously, the proposals were never buffered, which only produced correct signatures for requests without a body or requests to S3, ES or Glacier, which used the literal string
UNSIGNED-PAYLOAD. Buffering can now be disabled to use unsigned payloads with compatible services via the new
use_unsigned_payload filter option (default false).
Cache Filter and Cluster
The new version comes with serving HEAD requests from the cache. There is much appreciation due to the recent addition of the default value of 5 seconds for
DNS and DNS Cache
The new version disables the integration between ExtensionWithMatcher and HTTP filters by default to reflect its experimental status. We can enable this feature by setting envoy.reloadable_features.experimental_matching_api to true. Also, we can see for HTTP, the replaced setting
envoy.reloadable_features.strict_1xx_and_204_response_headers with settings
envoy.reloadable_features.require_strict_1xx_and_204_response_headers (require upstream 1xx or 204 responses to not have Transfer-Encoding or non-zero Content-Length headers) and
envoy.reloadable_features.send_strict_1xx_and_204_response_headers (do not send 1xx or 204 responses with these headers). Both are true by default. Now, HTTP will stop sending the transfer-encoding header for 304. We can temporarily revert this behaviour by setting
envoy.reloadable_features.no_chunked_encoding_header_for_304 to false. Again, the behaviour of the present_match in route header matcher changed. The behaviour ignored the value of
present_match in the past. The new behaviour now performs present_match when the value is true. Now, when the value is false, we perform an absent match.
With the update, we can define the connection balance config within the listener where the redirection of sockets occurs. We can clear that field to restore the previous behaviour. When balancing across active listeners and wildcards, the new version uses matching. The behaviour is changed to return the listener that matches the IP family type associated with the listener’s socket address. We can revert any unexpected behavioural changes by setting runtime guard
envoy.reloadable_features.listener_wildcard_match_ip_family to false.
TCP and UDP
For TCP, we can see switching to the new connection pool by default. We can revert any unexpected behavioural changes by setting runtime guard
envoy.reloadable_features.new_tcp_connection_pool to false. Now, the limit for each UDP listener to read maximum packets per event loop is 6000. We can again temporarily revert this behaviour by setting
envoy.reloadable_features.udp_per_event_loop_read_limit to false.
The bug fix changes are expected to improve the state of the world and are unlikely to have adverse effects
If we see the setting of
payload_passthrough to false, the downstream response content-type header will now be selected from the content-type entry in the JSON response’s headers map, if present.
The devs have fixed the cluster stats histograms in the new version by moving the accounting into the router filter. It means that we can now correctly compute the number of bytes sent and the previously ignored handling retries.
We can now see the fixing of double-counting of
server.days_until_first_cert_expiring during hot-restart. This stat was only incorrect until the parent process terminated.
We are again fixing of erroneous handling of invalid nghttp2 frames with the
NGHTTP2_ERR_REFUSED_STREAM error. Before the fix, Envoy would close the entire connection when nghttp2 triggered the null frame callback for the said error. This fixing will cause Envoy to terminate just the refused stream and retain the link. We can temporarily revert this behaviour by setting the
Envoy.reloadable_features.http2_consume_stream_refused_errors runtime guard to false. The port stripping now works for CONNECT requests, though restoring the port will occur if we send the CONNECT request upstream. We can again temporarily revert this behaviour by setting
Envoy.reloadable_features.strip_port_from_connect to false.
The unauthorized responses will now correctly include a www-authenticate header.
We see fixing a crash that could happen when a filter chain only listener update is followed by listener removal or a full listener update.
The new update fixed an issue that causes TAP sockets to panic during config validation mode.
We also see fixing the default sampling rate for AWS X-Ray tracer extension to be 5% instead of 50%.
The new version also fixed the timestamp serialization in annotations. A prior bug fix exposed an issue with timestamps which serialize as strings.
Removed Config or Runtime
All these things commonly occur at the end of the deprecation period. In case of event, we see the removal of
envoy.reloadable_features.activate_timers_next_event_loop runtime guard and legacy code path. Again for gzip, we can see removal of legacy HTTP Gzip filter and runtime guard
envoy.deprecated_features.allow_deprecated_gzip_http_filter. For HTTP, the new update removed
envoy.reloadable_features.allow_500_after_100 runtime guard and the legacy code path as well as
envoy.reloadable_features.always_apply_route_header_rules runtime guard and legacy code path. Also, we can see removal of
envoy.reloadable_features.hcm_stream_error_on_invalid_message for disabling closing HTTP/1.1 connections on error. we can still disable connection-closing by setting the HTTP/1 configuration
override_stream_error_on_invalid_http_message. We also see removal of
envoy.reloadable_features.http_set_copy_replace_all_headers runtime guard and legacy code paths. Again, for HTTP we can also see removal of
envoy.reloadable_features.overload_manager_disable_keepalive_drain_http2; Envoy will now always send GOAWAY to HTTP2 downstreams when the disable_keepalive overload action is active. Also, the removal of
envoy.reloadable_features.unify_grpc_handling runtime guard and legacy code paths. And lastly, in case of tls, we can see the removal of
envoy.reloadable_features.tls_use_io_handle_bio runtime guard and legacy code path.
The new version added the new response flag for overload manager termination. We will set the response flag when the overload manager terminates the HTTP stream.
Admission Control and Bandwidth Limit
A new feature like the addition of the
rps_threshold option is that when the average RPS of the sampling window is below that threshold, the filter will not throttle requests. Also, the addition of the
max_rejection_probability option to set an upper limit on the probability of rejection. A new HTTP bandwidth limit filter is a valuable addition.
For bootstrap, new features like the addition of
dns_resolution_config to aggregate all of the DNS resolver configuration in a single message os very useful. By setting no_default_search_domain to true, the DNS resolver will not use the default search domains. And also, by placing the resolvers, we can determine the external DNS servers they will use for external DNS queries.
The addition of
dns_resolution_config to aggregate all of the DNS resolver configuration in a single message will be there in the new version. By setting no_default_search_domain to true, the DNS resolver will not use the default search domains. Also, added
host_rewrite_literal to WeightedCluster will help users, and also the addition of a new feature such as
wait_for_warm_on_init will allow cluster readiness to not block on cluster warm-up. By default, we can confirm that this feature preserves existing behaviour. Currently, it is only applicable for DNS-based clusters.
Composite Filter, Config and Connection Limit
We can now use a composite filter with filters that also add an access logger, such as the WASM filter. The addition of stat
config_reload_time_ms is new, and also there is an addition of a new Network connection limit filter.
The restoration of crash context happens when processing requests or responses resulting from an asynchronous callback invokes a filter directly, unlike the call stacks that go through the various network layers to reach the filter eventually.
There is the addition of the
preresolve_hostnames option to the DNS cache config, which allows preresolving of hostnames into the cache upon cache creation. It might provide performance improvement, in the form of cache hits, for hostnames that we will see resolving during steady-state, and we will know it at config load time. Again, we can see the addition of the
dns_query_timeout option to the DNS cache config. This option allows explicitly controlling the timeout of underlying queries independently of the underlying DNS platform implementation. This timeout will lead to more deterministic DNS resolution times while getting coupled with success and failure retry policies.
In this new version, we can see the addition of the
DnsResolverOptions protobuf message to reconcile all of the DNS lookup option flags. By setting the configuration option
use_tcp_for_dns_lookups as accurate, we can make the underlying dns resolver library make only TCP queries to the DNS servers and set the configuration option
no_default_search_domain as accurate the DNS resolver library will not use the default search domains. The new version again comes with the addition of
DnsResolutionConfig to combine
dns_resolver_options and resolvers in a single protobuf message. We can now specify the field resolver with a list of DNS resolver addresses. If selected, the DNS client library will perform resolution via the underlying DNS resolvers. Otherwise, we will use the default system resolvers (e.g., /etc/resolv.conf).
The new update brings
dns_resolution_config to aggregate all of the DNS resolver configurations in a single message. By setting the configuration option
use_tcp_for_dns_lookups to true, we can make the dns filter’s external resolvers only answer queries using TCP. By selecting the configuration option
no_default_search_domain as accurate, the DNS resolver will not use the default search domains. And by setting the configuration resolvers, we can specify the external DNS servers for outer DNS queries, replacing the pre-existing alpha api field upstream_resolvers.
Dynamic Forward Proxy
With v1.19.0, we see the addition of the
dns_resolution_config option to the DNS cache config to aggregate all of the DNS resolver configuration in a single message. With setting one such configuration option
no_default_search_domain as accurate, the DNS resolver will not use the default search domains. And by setting the configuration resolvers, we can specify the external DNS servers for external DNS queries instead of the system default resolvers.
External Authorization Filter
With the new update, the devs brought the
bootstrap_metadata_labels_key option to configure labels of destination service.
New features in HTTP includes the addition of new field
extensions.filters.network.http_connection_manager.v3.HttpFilter. When set to true, Envoy will ignore the unsupported HTTP filters. It is also the same with an unsupported HTTP filter in the typed per filter config. Also, we can see scheme options for adding or overwriting schemes. Another exciting addition of stripping trailing host dot from host header support is now live, along with added support for original IP detection extensions. There is initial addition of two initial extensions, such as the custom header extension and the xff extension. Again, the devs have added a new option to upstream HTTP/2
keepalive to send a PING ahead of a new stream if the connection has been idle for a sufficient duration. We can also see the added ability to unescape slash sequences in the path. Requests with unescaped slashes can be proxied, rejected or redirected to the new unescaped path. By default, this feature is disabled. We can override the default behavior through
http_connection_manager.path_with_escaped_slashes_action runtime variable. We can selectively enable this action for a portion of requests by setting the
http_connection_manager.path_with_escaped_slashes_action_sampling runtime variable. The new addition of support for upstream and downstream alpha HTTP/3 is visible. We can refer to
quic_options for downstream and the new
http_protocol_options for upstream HTTP/3. Lastly, we can see the raising of the max configurable
max_request_headers_kb limit to 8192 KiB (8MiB) from 96 KiB in the HTTP connection manager.
With the new update comes a new input matcher that matches an IP address against a list of CIDR ranges.
The update now adds the ability to change an existing listener’s address and have added a filter chain match support for direct source address.
Local Rate Limit Filter
We can see added suppoort for locally rate limiting http requests on a per connection basis in the new release. We can enable this by setting the
local_rate_limit_per_downstream_connection field to true.
The new release brought added support for sending metric tags as labels. We can enable this by setting the
emit_tags_as_labels field to true.
The maintainers have given added support for generating the header while using the HTTP connection manager in the new version. We can do this by using the Proxy Protocol-Transport Socket on upstream clusters.
Request Without Query
The addition of access log formatter extension implementing command operator
REQ_WITHOUT_QUERY to log the request path while excluding the query string is a new feature.
Router, Stats and TCP
The added option
suppress_grpc_request_failure_code_stats to the router will now allow users to exclude incrementing HTTP status code stats on gRPC requests. Also, for stats, we can see the addition of native Graphite-formatted tag support. And in the case of TCP, the devs have given added support for preconnecting. Preconnecting is off by default, but we highly recommended it for clusters serving latency-sensitive traffic.
TLS, Tracing and UDP Proxy
In the new update, TLS will allow dual ECDSA/RSA certs via SDS. Earlier, SDS only supported a single certificate per context and only kept dual cert via non-SDS. In tracing, we can see the addition of an option to
use_request_id_for_trace_sampling, which allows configuring whether to perform sampling-based on
x-request-id or not. Also, for
udp_proxy, we have seen added key as another hash policy to support hash-based routing on any given key.
Windows Container Image
The new release brings a new user, EnvoyUser, part of the Network Configuration Operators group, to the container image.
Some of the things got deprecated too during this release. And, we will have a look at that too.
The new update brings deprecation of the field
known_suffixes. The internal data management of the filter has changed, and the filter no longer uses the
Dynamic Forward Proxy
Last but not least, the new version of Envoy is out now, and you can check it out by clicking here.