[PATCH] Mail: nginx_error:cache file has too long header

[Yongguang Bai]

# HG changeset patch
# User Yongguang Bai <***@quantil.com>
# Date 1543276278 28800
# Mon Nov 26 15:51:18 2018 -0800
# Node ID 4d48e1f3ebf6f0634e8e2cbd057d9fcfbea9d988
# Parent a7ff19afbb14795fef14f599a304d0ad21052b70
Mail: nginx_error:cache file has too long header

This error is printed when cached file is expired and the
response is changed in proxied server, and Nginx is under
heavy load.

diff -r a7ff19afbb14 -r 4d48e1f3ebf6 src/http/ngx_http_file_cache.c
--- a/src/http/ngx_http_file_cache.c Mon Nov 26 18:29:56 2018 +0300
+++ b/src/http/ngx_http_file_cache.c Mon Nov 26 15:51:18 2018 -0800
@@ -376,6 +376,10 @@
ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
"http file cache fd: %d", of.fd);

+ if(c->uniq != of.uniq){
+ c->body_start = ngx_pagesize;
+ }
+
c->file.fd = of.fd;
c->file.log = r->connection->log;
c->uniq = of.uniq;

[Maxim Dounin]

Hello!

Post by Yongguang Bai
# HG changeset patch
# Date 1543276278 28800
# Mon Nov 26 15:51:18 2018 -0800
# Node ID 4d48e1f3ebf6f0634e8e2cbd057d9fcfbea9d988
# Parent a7ff19afbb14795fef14f599a304d0ad21052b70
Mail: nginx_error:cache file has too long header
This error is printed when cached file is expired and the
response is changed in proxied server, and Nginx is under
heavy load.
diff -r a7ff19afbb14 -r 4d48e1f3ebf6 src/http/ngx_http_file_cache.c
--- a/src/http/ngx_http_file_cache.c Mon Nov 26 18:29:56 2018 +0300
+++ b/src/http/ngx_http_file_cache.c Mon Nov 26 15:51:18 2018 -0800
@@ -376,6 +376,10 @@
ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
"http file cache fd: %d", of.fd);
+ if(c->uniq != of.uniq){
+ c->body_start = ngx_pagesize;
+ }
+
c->file.fd = of.fd;
c->file.log = r->connection->log;
c->uniq = of.uniq;

This change looks wrong. In particular, ngx_pagesize, which is
usually 4k, can be smaller than existing c->body_start, and this
change will make things worse instead of fixing anything.

Also, could you please elaborate what exactly you are trying to
fix?

There is a known race which may lead to "cache file has too long
header" error - and, actually the error was added to ensure that
this race is properly handled if happens, see here:

http://hg.nginx.org/nginx/rev/6f97afc238de
http://mailman.nginx.org/pipermail/nginx-devel/2011-September/001287.html

Note that this race can only result in problems if for some reason
the backend returns responses with different headers nearly at the
same time. This may indicate that there is something wrong with
the backend.

Also, there is a known bug with multiple variants which also can
lead to the same error. The patch can be found here:

http://mailman.nginx.org/pipermail/nginx-devel/2018-January/010774.html

If you are hitting this bug, please test if the patch works for
you.

[Yongguang Bai]

Hi Maxim,

Sorry for my late reply.

We use Nginx as CDN Cache. And it is normal case that the responses are
different from different origin servers. For instance: the origin server
could be
the customer server or some other CDN provider Cache server. The response
header could be different.

In race condition, I found the inconsistency between disk file and metadata
in
shared memory.
When (c->uniq != of.uniq) happens, I saw " too long header" in error log.

To fix this, I picked the ngx_pagesize to initiate c->body_start again.
then the "too long header"
is gone from error log.

I will try your patches next week. Thanks!

Post by Maxim Dounin
Hello!

Post by Yongguang Bai
# HG changeset patch
# Date 1543276278 28800
# Mon Nov 26 15:51:18 2018 -0800
# Node ID 4d48e1f3ebf6f0634e8e2cbd057d9fcfbea9d988
# Parent a7ff19afbb14795fef14f599a304d0ad21052b70
Mail: nginx_error:cache file has too long header
This error is printed when cached file is expired and the
response is changed in proxied server, and Nginx is under
heavy load.
diff -r a7ff19afbb14 -r 4d48e1f3ebf6 src/http/ngx_http_file_cache.c
--- a/src/http/ngx_http_file_cache.c Mon Nov 26 18:29:56 2018 +0300
+++ b/src/http/ngx_http_file_cache.c Mon Nov 26 15:51:18 2018 -0800
@@ -376,6 +376,10 @@
ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
"http file cache fd: %d", of.fd);
+ if(c->uniq != of.uniq){
+ c->body_start = ngx_pagesize;
+ }
+
c->file.fd = of.fd;
c->file.log = r->connection->log;
c->uniq = of.uniq;

This change looks wrong. In particular, ngx_pagesize, which is
usually 4k, can be smaller than existing c->body_start, and this
change will make things worse instead of fixing anything.
Also, could you please elaborate what exactly you are trying to
fix?
There is a known race which may lead to "cache file has too long
header" error - and, actually the error was added to ensure that
http://hg.nginx.org/nginx/rev/6f97afc238de
http://mailman.nginx.org/pipermail/nginx-devel/2011-September/001287.html
Note that this race can only result in problems if for some reason
the backend returns responses with different headers nearly at the
same time. This may indicate that there is something wrong with
the backend.
Also, there is a known bug with multiple variants which also can
http://mailman.nginx.org/pipermail/nginx-devel/2018-January/010774.html
If you are hitting this bug, please test if the patch works for
you.
--
Maxim Dounin
http://mdounin.ru/

--
Best Regards,

*Yongguang Bai*

Engineering Manager
[image: Quantil]

Connecting users with content...it's that simple.

Direct: [phone]
Office: [phone]

[image: Tweeter] <https://twitter.com/Team_Quantil> [image: Google Plus]
<https://plus.google.com/+Quantil_team/> [image: Linked In]
<https://www.linkedin.com/company/quantil>