v14 Crashing Issue + Active4D

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

v14 Crashing Issue + Active4D

Peter Jakobsson-2
I’ve been researching an apparent bug for nearly a week now which is quite difficult to pin down.

It occurs with the Active4D demo structure running on 4D Client Windows and is entirely reproducible but not with a fresh structure and I can’t quite see why. (Note: It’s not just the demo, it occurs with any Active4D structure).

As far as I can see, the crash occurs when re-activating a web process that has “timed out” when 4D Client’s “Reuse temporary contexts” property is switched on.

For example:

[1] - set the number of web processes to around 10 (it also crashes with 100 but I use 10 for control)

[2] - set the “Re-use temporary contexts” to on and the web process timeout to 5 minutes ( <— that bit is important)

[3] - load the Active4D demo with enough requests over a short space of time to “activate” most or all of the web processes (usually rapid-fire refresh with the browser for about a minute does it)

[4] - wait 6 minutes for the “temporary contexts” to time out ( <— note, not to be confused with 4D’s old ‘context mode’. Just means web process)

[5] - do one single refresh with the browser. The web server will now crash.

However when I set this up in a brand new structure that isn’t an upgrade (the Active4D demo and my production databases were both upgrades) it doesn’t happen.

I still have lots of testing to do to further narrow things down but as this is all a massively time consuming process, I’d appreciate any input from anyone that has noticed this phenomenon. Otherwise I’ll report back when I’ve made more progress.

Regards

Peter

Active4D 6.1r8
4D 14.3 Beta (Build 172314)
Windows 7 (4D Server)
Windows 2008 Server R2 (4D Client)


_______________________________________________
Active4D-dev mailing list
[hidden email]
http://list.aparajitaworld.com/listinfo/active4d-dev
Archives: http://active4d-nabble.aparajitaworld.com/
Reply | Threaded
Open this post in threaded view
|

Re: v14 Crashing Issue + Active4D

aparajita
Administrator
> However when I set this up in a brand new structure that isn’t an upgrade (the Active4D demo and my production databases were both upgrades) it doesn’t happen.

It sounds like there is a web server setting that differs. Check every setting in the new structure vs. an upgrade.

Regards,

  Aparajita

_______________________________________________
Active4D-dev mailing list
[hidden email]
http://list.aparajitaworld.com/listinfo/active4d-dev
Archives: http://active4d-nabble.aparajitaworld.com/
Reply | Threaded
Open this post in threaded view
|

Re: v14 Crashing Issue + Active4D

Peter Jakobsson-2

On 5 Nov 2014, at 09:56, Aparajita Fishman <[hidden email]> wrote:

> It sounds like there is a web server setting that differs. Check every setting in the new structure vs. an upgrade.

Just to update. It’s nothing to do with Active4D it would appear.

I can reproduce the crashing without the call to A4D Execute Request as long as “On Web Connection” is called.

It does appear to affect upgraded structures distinctly from vanilla v14 ones but I need to confirm this.

The “jerky” requests syndrome I described last week also appear to be related (even after removing the “close connection” code from the shell) - the vanilla v14 web serving is silky smooth with no jerky requests and no crashing with expired web processes. The upgraded structure suffers from both.

Now that I know I can reproduce it with a few lines of non Active4D code in “On Web Connection” I’ll create a vanilla v11 structure, upgrade it and see if that exhibits the problem distinctly from an identical native v14 structure.

Peter

_______________________________________________
Active4D-dev mailing list
[hidden email]
http://list.aparajitaworld.com/listinfo/active4d-dev
Archives: http://active4d-nabble.aparajitaworld.com/
Reply | Threaded
Open this post in threaded view
|

Re: v14 Crashing Issue + Active4D

Peter Schumacher-2
Hi Peter,

Just the obvious... did you run a MSC check on the updated structure lately?

Greetings,
Peter

 
_______________________________________________
Active4D-dev mailing list
[hidden email]
http://list.aparajitaworld.com/listinfo/active4d-dev
Archives: http://active4d-nabble.aparajitaworld.com/
Reply | Threaded
Open this post in threaded view
|

Re: v14 Crashing Issue + Active4D

Peter Jakobsson-2
On 5 Nov 2014, at 19:02, Peter Schumacher <[hidden email]> wrote:

> Just the obvious... did you run a MSC check on the updated structure lately?

Nope - of course not.

That would be too sensible for me :)

P

_______________________________________________
Active4D-dev mailing list
[hidden email]
http://list.aparajitaworld.com/listinfo/active4d-dev
Archives: http://active4d-nabble.aparajitaworld.com/
Reply | Threaded
Open this post in threaded view
|

Re: v14 Crashing Issue (Detected)

Peter Jakobsson-2

Hi

I seem to have got to the bottom of this. The issue is with v13’s new “Automatic Session Management” property. With this switched on, performance is amazingly silky smooth and stable (compared with v12). With “Automatic Session Management” off, it is jerky and unstable.

In particular, 2 adverse conditions prevail when this feature is switched off:

[1] - under moderate loads, every 4th or 5th request “locks up”

[2] - if the “Reuse Temporary Contexts (in remote mode)” feature is active and traffic dwindles to the point that some web processes expire due to timeout, the web server will crash when it next receives a request

Just so happens that this exactly corresponds to a typical Active4D configuration (since we’ve had automatic session management since the dinosaurs walked the earth).

In upgraded databases, the feature is off by default. In v14 native ones, it’s on by default which explains why upgraded ones exhibit the acute condition. In particular:

**** Upgraded Structure Defaults *****

Auto Session Man: Off
Web Process Timeout: 5 minutes - 1 hour

**** v14 New Structure Defaults *****

Auto Session Man: On
Web Process Timeout: 8 hours

Hopefully this can get addressed before 14.3 is out. I’ll report it on the beta forum.

Peter

_______________________________________________
Active4D-dev mailing list
[hidden email]
http://list.aparajitaworld.com/listinfo/active4d-dev
Archives: http://active4d-nabble.aparajitaworld.com/
Reply | Threaded
Open this post in threaded view
|

Re: v14 Crashing Issue (Detected)

Peter Schumacher-2
Hi Peter,

Strange - "automatic session handling" and "reuse context (4D keeps the values of the variables, good for testing)" are the first things I turn off. Same for "keep a live", I turn it off.

I'd create some spare web processes (there is a DB settings for this). You didn't lower the stack size anywhere, did you? If you are using nightly builds, use a fresh more recent one to avoid the funny web things which popped up lately.

Greetings,
Peter
  _____  

From: Peter Jakobsson [mailto:[hidden email]]
To: Active4D Developer Discussion List [mailto:[hidden email]]
Sent: Wed, 05 Nov 2014 22:47:14 +0100
Subject: Re: [Active4d-dev] v14 Crashing Issue (Detected)


  Hi
 
  I seem to have got to the bottom of this. The issue is with v13’s new “Automatic Session Management” property. With this switched on, performance is amazingly silky smooth and stable (compared with v12). With “Automatic Session Management” off, it is jerky and unstable.
 
  In particular, 2 adverse conditions prevail when this feature is switched off:
 
  [1] - under moderate loads, every 4th or 5th request “locks up”
 
  [2] - if the “Reuse Temporary Contexts (in remote mode)” feature is active and traffic dwindles to the point that some web processes expire due to timeout, the web server will crash when it next receives a request
 
  Just so happens that this exactly corresponds to a typical Active4D configuration (since we’ve had automatic session management since the dinosaurs walked the earth).
 
  In upgraded databases, the feature is off by default. In v14 native ones, it’s on by default which explains why upgraded ones exhibit the acute condition. In particular:
 
  **** Upgraded Structure Defaults *****
 
  Auto Session Man: Off
  Web Process Timeout: 5 minutes - 1 hour
 
  **** v14 New Structure Defaults *****
 
  Auto Session Man: On
  Web Process Timeout: 8 hours
 
  Hopefully this can get addressed before 14.3 is out. I’ll report it on the beta forum.
 
  Peter
 
  _______________________________________________
  Active4D-dev mailing list
  [hidden email]
  http://list.aparajitaworld.com/listinfo/active4d-dev
  Archives: http://active4d-nabble.aparajitaworld.com/
   
_______________________________________________
Active4D-dev mailing list
[hidden email]
http://list.aparajitaworld.com/listinfo/active4d-dev
Archives: http://active4d-nabble.aparajitaworld.com/
Reply | Threaded
Open this post in threaded view
|

Re: v14 Crashing Issue (Detected)

Peter Jakobsson-2

On 5 Nov 2014, at 22:57, Peter Schumacher <[hidden email]> wrote:

> Strange - "automatic session handling" and "reuse context (4D keeps the values of the variables, good for testing)" are the first things I turn off. Same for "keep a live", I turn it off.

Hi Peter

If you run with “Reuse temporary contexts” off then that explains why you don’t see the problem (at least according to my observations).

I’ve always run with it on because I’ve found that 4D Client runs much smoother - you don’t get a microsecond delay while it starts up a new process.

> I'd create some spare web processes (there is a DB settings for this). You didn't lower the stack size anywhere, did you? If you are using nightly builds, use a fresh more recent one to avoid the funny web things which popped up lately.


The web processes are set to around 100 I think but will check. I don’t think they’re over taxed. I’ll check the latest fix list as you suggest (I tend to monitor your announcements as opposed to 4D Forum as it’s easier).

Peter

_______________________________________________
Active4D-dev mailing list
[hidden email]
http://list.aparajitaworld.com/listinfo/active4d-dev
Archives: http://active4d-nabble.aparajitaworld.com/
Reply | Threaded
Open this post in threaded view
|

Re: v14 Crashing Issue (Detected)

Peter Jakobsson-2
In reply to this post by Peter Schumacher-2
On 5 Nov 2014, at 22:57, Peter Schumacher <[hidden email]> wrote:

>  If you are using nightly builds, use a fresh more recent one to avoid the funny web things which popped up lately.

I must say this one stood out slightly:
ACI0090150 : [129865] When creating more than 32,767 windows, 4D freezes
No kidding !

I’ll sure sleep sounder now I know that's fixed :)

P

_______________________________________________
Active4D-dev mailing list
[hidden email]
http://list.aparajitaworld.com/listinfo/active4d-dev
Archives: http://active4d-nabble.aparajitaworld.com/
Reply | Threaded
Open this post in threaded view
|

Re: v14 Crashing Issue (Detected)

Peter Schumacher-2
In reply to this post by Peter Jakobsson-2
Hi Peter,

100 is far too low. Set them to 200 or more. How many web process do you allow at all? Set the upper limit to 2000, doesn't cost you any if never used.

I'm asking because of the "keep a live". I don't know your timeout settings. Imagine lots of traffic and the connections are kept open (a live) and 4D hits the ceiling of your total amount of web processes.

Greetings,
Peter
  _____  

From: Peter Jakobsson [mailto:[hidden email]]
To: Active4D Developer Discussion List [mailto:[hidden email]]
Sent: Wed, 05 Nov 2014 23:16:18 +0100
Subject: Re: [Active4d-dev] v14 Crashing Issue (Detected)


  On 5 Nov 2014, at 22:57, Peter Schumacher <[hidden email]> wrote:
 
  > Strange - "automatic session handling" and "reuse context (4D keeps the values of the variables, good for testing)" are the first things I turn off. Same for "keep a live", I turn it off.
 
  Hi Peter
 
  If you run with “Reuse temporary contexts” off then that explains why you don’t see the problem (at least according to my observations).
 
  I’ve always run with it on because I’ve found that 4D Client runs much smoother - you don’t get a microsecond delay while it starts up a new process.
 
  > I'd create some spare web processes (there is a DB settings for this). You didn't lower the stack size anywhere, did you? If you are using nightly builds, use a fresh more recent one to avoid the funny web things which popped up lately.
 
 
  The web processes are set to around 100 I think but will check. I don’t think they’re over taxed. I’ll check the latest fix list as you suggest (I tend to monitor your announcements as opposed to 4D Forum as it’s easier).
 
  Peter
 
  _______________________________________________
  Active4D-dev mailing list
  [hidden email]
  http://list.aparajitaworld.com/listinfo/active4d-dev
  Archives: http://active4d-nabble.aparajitaworld.com/
   
_______________________________________________
Active4D-dev mailing list
[hidden email]
http://list.aparajitaworld.com/listinfo/active4d-dev
Archives: http://active4d-nabble.aparajitaworld.com/
Reply | Threaded
Open this post in threaded view
|

Re: v14 Crashing Issue (Detected)

Peter Jakobsson-2

On 6 Nov 2014, at 00:16, Peter Schumacher <[hidden email]> wrote:

> 100 is far too low. Set them to 200 or more

That’s strange. You think 100 is to low?

We never get more than about 30-40 processes in use as far as I know - if that. The web system is load balanced across 3 4D Client machines but even when it was only 1 I was never aware of more than 20 to 30 web processes. (That was from monitoring in runtime explorer years ago. v14 doesn’t seem to show the web processes though).

Are you sure you’re web server’s using all those processes ?

> I'm asking because of the "keep a live". I don't know your timeout settings. Imagine lots of traffic and the connections are kept open (a live) and 4D hits the ceiling of your total amount of web processes.


I think we have “keep alive” on - but there are 2 types of “keep alive”. 1 is the “Temporary Contexts” property which basically just keeps the web process running even if it’s not being used, the other is the http “Keep Alive” property. Is that the one you mean ?

The web processes time out after 5 minutes and I think http “Keep Alive” is on.

Just to clarify, how I tested the crashing. I used a kitchen digital egg timer to determine the exact moment when the web processes timed out (It’s about the same, depending on how you like your eggs). If I refreshed the browser at 4:45 and did some browsing, the web server would stay up. If I refreshed at 5:15 it would instantly crash with just1 request. I repeated this pattern many times and with 4 structures:

 - an upgraded production structure
 - the upgraded Active4D demo
 - a v14 virgin structure
 - a v14 virgin structure with migrated Active4D demo code

All with consistent results. Stay up if the web process (“Temporary Context” as 4D calls it) hadn’t timed out. Crash if it had timed out. But “Auto Session Management” has to be off.

Peter

_______________________________________________
Active4D-dev mailing list
[hidden email]
http://list.aparajitaworld.com/listinfo/active4d-dev
Archives: http://active4d-nabble.aparajitaworld.com/
Reply | Threaded
Open this post in threaded view
|

Re: v14 Crashing Issue (Detected)

Peter Jakobsson-2
In reply to this post by Peter Schumacher-2

On 6 Nov 2014, at 00:16, Peter Schumacher <[hidden email]> wrote:

> 100 is far too low. Set them to 200 or more

By the way. When I looked into this a few years ago, the reason I concluded that 4D’s web processes never got up to a significant amount under high loading was that the requests were being rejected at the socket layer before they had a chance to kick off any new processes.

i.e. 4D wasn’t able to generate new processes fast enough to absorb the socket backlog in the TCP layer. NTK has a feature that allows you to “tune” this. You can increase the size of the socket backlog queue and that lets 4D absorb a slightly higher load before the queue saturates.

The last few days I’ve been working with v14, it feels completely different. There are few or no socket layer rejections which allows the web server (“On Web Connection”) to absorb a much higher load. It’s basically feels like having Apache in front of the old web server. I think this one looks very promising if they can get it stable.

Maybe that accounts for your much higher process count - I was working with v12 before, not v13 which is the version they itroduced the new web engine.

Peter

_______________________________________________
Active4D-dev mailing list
[hidden email]
http://list.aparajitaworld.com/listinfo/active4d-dev
Archives: http://active4d-nabble.aparajitaworld.com/