Active4D Slowdown then Stall (weird A4D_HTTPD_RequestHandler_* behavior)

classic Classic list List threaded Threaded
28 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Active4D Slowdown then Stall (weird A4D_HTTPD_RequestHandler_* behavior)

Perkins, Bradley D
We upgraded a long running A4D-based system late last week and are noticing that Active4D will eventually become unresponsive. We can't serve for more than a day and web apps have been unresponsive the last two mornings. The old system would run for a long time without fail.

We went from
4D 14.2 to 4D15.4
Active4D running on OS 10.11.latest to OS 10.12.4 (Sierra)
Active4D is behind Apache. The Mac is a "client". Server runs on Windows.
Replaced a 6+ year old Xserve with a Mac Mini (16 GB RAM)
Upgraded to latest Active4D and latest NTK w/ necessary changes to Shell Code.

Admittedly a lot of changes, but this update was mandated by institutional cybersecurity group (Sierra was required). Sierra would not install on the Xserves. 15.4 is the only version close to certified for Sierra.

When this occurs I find:
(a) Apache can still serve static content fine.
(b) 4D is still running and responsive to user input.
(c) I can see many A4D_HTTPD_RequestHandler_* processes on the server. Many are of type "4D Client Database Process" with num "0" and State "Running". That doesn't seem right to me.
I'm currently observing both our production and development servers now and neither show any "4D Client Database Process" processes. I can force the creation of new processes by using ApacheBench to make many concurrent calls. All of the new process are client processes.

Can anyone suggest how those  "4D Client Database Process" are being created?

Thanks,

Brad Perkins




_______________________________________________
Active4D-dev mailing list
[hidden email]
http://list.aparajitaworld.com/listinfo/active4d-dev
Archives: http://active4d-nabble.aparajitaworld.com/
Reply | Threaded
Open this post in threaded view
|

Re: Active4D Slowdown then Stall (weird A4D_HTTPD_RequestHandler_* behavior)

Perkins, Bradley D
I noticed something else:

On the production server the A4D_HouseKeeper process was of type "4D Client Database Process" yet on development it was type "4D Client Process".
When I relaunched the production server there is no A4D_HouseKeeper at all.

Brad

From: Brad Perkins <[hidden email]<mailto:[hidden email]>>
Date: Wednesday, April 19, 2017 at 12:16 PM
To: "[hidden email]<mailto:[hidden email]>" <[hidden email]<mailto:[hidden email]>>
Subject: Active4D Slowdown then Stall (weird A4D_HTTPD_RequestHandler_* behavior)

We upgraded a long running A4D-based system late last week and are noticing that Active4D will eventually become unresponsive. We can't serve for more than a day and web apps have been unresponsive the last two mornings. The old system would run for a long time without fail.

We went from
4D 14.2 to 4D15.4
Active4D running on OS 10.11.latest to OS 10.12.4 (Sierra)
Active4D is behind Apache. The Mac is a "client". Server runs on Windows.
Replaced a 6+ year old Xserve with a Mac Mini (16 GB RAM)
Upgraded to latest Active4D and latest NTK w/ necessary changes to Shell Code.

Admittedly a lot of changes, but this update was mandated by institutional cybersecurity group (Sierra was required). Sierra would not install on the Xserves. 15.4 is the only version close to certified for Sierra.

When this occurs I find:
(a) Apache can still serve static content fine.
(b) 4D is still running and responsive to user input.
(c) I can see many A4D_HTTPD_RequestHandler_* processes on the server. Many are of type "4D Client Database Process" with num "0" and State "Running". That doesn't seem right to me.
I'm currently observing both our production and development servers now and neither show any "4D Client Database Process" processes. I can force the creation of new processes by using ApacheBench to make many concurrent calls. All of the new process are client processes.

Can anyone suggest how those  "4D Client Database Process" are being created?

Thanks,

Brad Perkins




_______________________________________________
Active4D-dev mailing list
[hidden email]
http://list.aparajitaworld.com/listinfo/active4d-dev
Archives: http://active4d-nabble.aparajitaworld.com/
Reply | Threaded
Open this post in threaded view
|

Re: Active4D Slowdown then Stall (weird A4D_HTTPD_RequestHandler_* behavior)

Aparajita Fishman
Just to be clear, the Mac Mini is a client that is the Active4D web server?

Have you turned on Active4D debug logging and looked for anything suspicious in the log?

When you say that many A4D_HTTPD_RequestHandler processes were on the server, you mean the Mac Mini or the Windows server?

Did you update the A4D_NTKConfig list?

All the best,

  - Aparajita

> On Apr 19, 2017, at 11:30 AM, Perkins, Bradley D <[hidden email]> wrote:
>
> I noticed something else:
>
> On the production server the A4D_HouseKeeper process was of type "4D Client Database Process" yet on development it was type "4D Client Process".
> When I relaunched the production server there is no A4D_HouseKeeper at all.
>
> Brad
>
> From: Brad Perkins <[hidden email]<mailto:[hidden email]>>
> Date: Wednesday, April 19, 2017 at 12:16 PM
> To: "[hidden email]<mailto:[hidden email]>" <[hidden email]<mailto:[hidden email]>>
> Subject: Active4D Slowdown then Stall (weird A4D_HTTPD_RequestHandler_* behavior)
>
> We upgraded a long running A4D-based system late last week and are noticing that Active4D will eventually become unresponsive. We can't serve for more than a day and web apps have been unresponsive the last two mornings. The old system would run for a long time without fail.
>
> We went from
> 4D 14.2 to 4D15.4
> Active4D running on OS 10.11.latest to OS 10.12.4 (Sierra)
> Active4D is behind Apache. The Mac is a "client". Server runs on Windows.
> Replaced a 6+ year old Xserve with a Mac Mini (16 GB RAM)
> Upgraded to latest Active4D and latest NTK w/ necessary changes to Shell Code.
>
> Admittedly a lot of changes, but this update was mandated by institutional cybersecurity group (Sierra was required). Sierra would not install on the Xserves. 15.4 is the only version close to certified for Sierra.
>
> When this occurs I find:
> (a) Apache can still serve static content fine.
> (b) 4D is still running and responsive to user input.
> (c) I can see many A4D_HTTPD_RequestHandler_* processes on the server. Many are of type "4D Client Database Process" with num "0" and State "Running". That doesn't seem right to me.
> I'm currently observing both our production and development servers now and neither show any "4D Client Database Process" processes. I can force the creation of new processes by using ApacheBench to make many concurrent calls. All of the new process are client processes.
>
> Can anyone suggest how those  "4D Client Database Process" are being created?
>
> Thanks,
>
> Brad Perkins
>
>
>
>
> _______________________________________________
> Active4D-dev mailing list
> [hidden email]
> http://list.aparajitaworld.com/listinfo/active4d-dev
> Archives: http://active4d-nabble.aparajitaworld.com/

_______________________________________________
Active4D-dev mailing list
[hidden email]
http://list.aparajitaworld.com/listinfo/active4d-dev
Archives: http://active4d-nabble.aparajitaworld.com/
Reply | Threaded
Open this post in threaded view
|

Re: Active4D Slowdown then Stall (weird A4D_HTTPD_RequestHandler_* behavior)

Perkins, Bradley D
Yes, the Mac Mini is the client that is the Active4D web server. Initially
we were having 'sleep' problems caused by Apple's built in power saving
settings. I've resolved those.

I just arrived at work and can see that problems started ~4 am local time.
It is still down. I do have debug logging on and am going to look at those
shortly. I'll send you anything notable. I also want to see if I can
correlate any problems with incoming HTTPD requests from the Apache logs.

The many A4D_HTTPD_RequestHandler processes were observed on the Windows
server. With the way our web client user interface is programmed it is
difficult to get to the Runtime environment but I'm going to try that too.

re: Did you update the A4D_NTKConfig list? I had another developer work on
this. I'll need to find out if he did.

Thanks,

Brad

On 4/19/17, 5:34 PM, "Active4D-dev on behalf of Aparajita Fishman"
<[hidden email] on behalf of
[hidden email]> wrote:

>Just to be clear, the Mac Mini is a client that is the Active4D web
>server?
>
>Have you turned on Active4D debug logging and looked for anything
>suspicious in the log?
>
>When you say that many A4D_HTTPD_RequestHandler processes were on the
>server, you mean the Mac Mini or the Windows server?
>
>Did you update the A4D_NTKConfig list?
>
>All the best,
>
>  - Aparajita
>
>> On Apr 19, 2017, at 11:30 AM, Perkins, Bradley D <[hidden email]>
>>wrote:
>>
>> I noticed something else:
>>
>> On the production server the A4D_HouseKeeper process was of type "4D
>>Client Database Process" yet on development it was type "4D Client
>>Process".
>> When I relaunched the production server there is no A4D_HouseKeeper at
>>all.
>>
>> Brad
>>
>> From: Brad Perkins <[hidden email]<mailto:[hidden email]>>
>> Date: Wednesday, April 19, 2017 at 12:16 PM
>> To:
>>"[hidden email]<mailto:[hidden email]>"
>>
>><[hidden email]<mailto:[hidden email]>>
>> Subject: Active4D Slowdown then Stall (weird A4D_HTTPD_RequestHandler_*
>>behavior)
>>
>> We upgraded a long running A4D-based system late last week and are
>>noticing that Active4D will eventually become unresponsive. We can't
>>serve for more than a day and web apps have been unresponsive the last
>>two mornings. The old system would run for a long time without fail.
>>
>> We went from
>> 4D 14.2 to 4D15.4
>> Active4D running on OS 10.11.latest to OS 10.12.4 (Sierra)
>> Active4D is behind Apache. The Mac is a "client". Server runs on
>>Windows.
>> Replaced a 6+ year old Xserve with a Mac Mini (16 GB RAM)
>> Upgraded to latest Active4D and latest NTK w/ necessary changes to
>>Shell Code.
>>
>> Admittedly a lot of changes, but this update was mandated by
>>institutional cybersecurity group (Sierra was required). Sierra would
>>not install on the Xserves. 15.4 is the only version close to certified
>>for Sierra.
>>
>> When this occurs I find:
>> (a) Apache can still serve static content fine.
>> (b) 4D is still running and responsive to user input.
>> (c) I can see many A4D_HTTPD_RequestHandler_* processes on the server.
>>Many are of type "4D Client Database Process" with num "0" and State
>>"Running". That doesn't seem right to me.
>> I'm currently observing both our production and development servers now
>>and neither show any "4D Client Database Process" processes. I can force
>>the creation of new processes by using ApacheBench to make many
>>concurrent calls. All of the new process are client processes.
>>
>> Can anyone suggest how those  "4D Client Database Process" are being
>>created?
>>
>> Thanks,
>>
>> Brad Perkins
>>
>>
>>
>>
>> _______________________________________________
>> Active4D-dev mailing list
>> [hidden email]
>> http://list.aparajitaworld.com/listinfo/active4d-dev
>> Archives: http://active4d-nabble.aparajitaworld.com/
>
>_______________________________________________
>Active4D-dev mailing list
>[hidden email]
>http://list.aparajitaworld.com/listinfo/active4d-dev
>Archives: http://active4d-nabble.aparajitaworld.com/

_______________________________________________
Active4D-dev mailing list
[hidden email]
http://list.aparajitaworld.com/listinfo/active4d-dev
Archives: http://active4d-nabble.aparajitaworld.com/
Reply | Threaded
Open this post in threaded view
|

Re: Active4D Slowdown then Stall (weird A4D_HTTPD_RequestHandler_* behavior)

Aparajita Fishman
What volume of requests are you getting before it stalls?

You shouldn’t see the A4D_HTTPD_RequestHandler processes on the Windows server at all. They should only be on the client.

All the best,

  - Aparajita

> On Apr 20, 2017, at 7:09 AM, Perkins, Bradley D <[hidden email]> wrote:
>
> Yes, the Mac Mini is the client that is the Active4D web server. Initially
> we were having 'sleep' problems caused by Apple's built in power saving
> settings. I've resolved those.
>
> I just arrived at work and can see that problems started ~4 am local time.
> It is still down. I do have debug logging on and am going to look at those
> shortly. I'll send you anything notable. I also want to see if I can
> correlate any problems with incoming HTTPD requests from the Apache logs.
>
> The many A4D_HTTPD_RequestHandler processes were observed on the Windows
> server. With the way our web client user interface is programmed it is
> difficult to get to the Runtime environment but I'm going to try that too.
>
> re: Did you update the A4D_NTKConfig list? I had another developer work on
> this. I'll need to find out if he did.
>
> Thanks,
>
> Brad
>
> On 4/19/17, 5:34 PM, "Active4D-dev on behalf of Aparajita Fishman"
> <[hidden email] on behalf of
> [hidden email]> wrote:
>
>> Just to be clear, the Mac Mini is a client that is the Active4D web
>> server?
>>
>> Have you turned on Active4D debug logging and looked for anything
>> suspicious in the log?
>>
>> When you say that many A4D_HTTPD_RequestHandler processes were on the
>> server, you mean the Mac Mini or the Windows server?
>>
>> Did you update the A4D_NTKConfig list?
>>
>> All the best,
>>
>> - Aparajita
>>
>>> On Apr 19, 2017, at 11:30 AM, Perkins, Bradley D <[hidden email]>
>>> wrote:
>>>
>>> I noticed something else:
>>>
>>> On the production server the A4D_HouseKeeper process was of type "4D
>>> Client Database Process" yet on development it was type "4D Client
>>> Process".
>>> When I relaunched the production server there is no A4D_HouseKeeper at
>>> all.
>>>
>>> Brad
>>>
>>> From: Brad Perkins <[hidden email]<mailto:[hidden email]>>
>>> Date: Wednesday, April 19, 2017 at 12:16 PM
>>> To:
>>> "[hidden email]<mailto:[hidden email]>"
>>>
>>> <[hidden email]<mailto:[hidden email]>>
>>> Subject: Active4D Slowdown then Stall (weird A4D_HTTPD_RequestHandler_*
>>> behavior)
>>>
>>> We upgraded a long running A4D-based system late last week and are
>>> noticing that Active4D will eventually become unresponsive. We can't
>>> serve for more than a day and web apps have been unresponsive the last
>>> two mornings. The old system would run for a long time without fail.
>>>
>>> We went from
>>> 4D 14.2 to 4D15.4
>>> Active4D running on OS 10.11.latest to OS 10.12.4 (Sierra)
>>> Active4D is behind Apache. The Mac is a "client". Server runs on
>>> Windows.
>>> Replaced a 6+ year old Xserve with a Mac Mini (16 GB RAM)
>>> Upgraded to latest Active4D and latest NTK w/ necessary changes to
>>> Shell Code.
>>>
>>> Admittedly a lot of changes, but this update was mandated by
>>> institutional cybersecurity group (Sierra was required). Sierra would
>>> not install on the Xserves. 15.4 is the only version close to certified
>>> for Sierra.
>>>
>>> When this occurs I find:
>>> (a) Apache can still serve static content fine.
>>> (b) 4D is still running and responsive to user input.
>>> (c) I can see many A4D_HTTPD_RequestHandler_* processes on the server.
>>> Many are of type "4D Client Database Process" with num "0" and State
>>> "Running". That doesn't seem right to me.
>>> I'm currently observing both our production and development servers now
>>> and neither show any "4D Client Database Process" processes. I can force
>>> the creation of new processes by using ApacheBench to make many
>>> concurrent calls. All of the new process are client processes.
>>>
>>> Can anyone suggest how those  "4D Client Database Process" are being
>>> created?
>>>
>>> Thanks,
>>>
>>> Brad Perkins
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Active4D-dev mailing list
>>> [hidden email]
>>> http://list.aparajitaworld.com/listinfo/active4d-dev
>>> Archives: http://active4d-nabble.aparajitaworld.com/
>>
>> _______________________________________________
>> Active4D-dev mailing list
>> [hidden email]
>> http://list.aparajitaworld.com/listinfo/active4d-dev
>> Archives: http://active4d-nabble.aparajitaworld.com/
>
> _______________________________________________
> Active4D-dev mailing list
> [hidden email]
> http://list.aparajitaworld.com/listinfo/active4d-dev
> Archives: http://active4d-nabble.aparajitaworld.com/


_______________________________________________
Active4D-dev mailing list
[hidden email]
http://list.aparajitaworld.com/listinfo/active4d-dev
Archives: http://active4d-nabble.aparajitaworld.com/
Reply | Threaded
Open this post in threaded view
|

Re: Active4D Slowdown then Stall (weird A4D_HTTPD_RequestHandler_* behavior)

Perkins, Bradley D
I'll need to calculate those from the apache logs, but I'd say it isn't
extreme.

I'll send you screen shots of both client runtime explorer and windows
server to show what I'm seeing.

Also nothing notable in terms of errors in the active4D logs, but I'll
send that privately as well.

Thanks,

-- Brad

On 4/20/17, 8:48 AM, "Active4D-dev on behalf of Aparajita Fishman"
<[hidden email] on behalf of
[hidden email]> wrote:

>What volume of requests are you getting before it stalls?
>
>You shouldn’t see the A4D_HTTPD_RequestHandler processes on the Windows
>server at all. They should only be on the client.
>
>All the best,
>
>  - Aparajita
>
>> On Apr 20, 2017, at 7:09 AM, Perkins, Bradley D <[hidden email]>
>>wrote:
>>
>> Yes, the Mac Mini is the client that is the Active4D web server.
>>Initially
>> we were having 'sleep' problems caused by Apple's built in power saving
>> settings. I've resolved those.
>>
>> I just arrived at work and can see that problems started ~4 am local
>>time.
>> It is still down. I do have debug logging on and am going to look at
>>those
>> shortly. I'll send you anything notable. I also want to see if I can
>> correlate any problems with incoming HTTPD requests from the Apache
>>logs.
>>
>> The many A4D_HTTPD_RequestHandler processes were observed on the Windows
>> server. With the way our web client user interface is programmed it is
>> difficult to get to the Runtime environment but I'm going to try that
>>too.
>>
>> re: Did you update the A4D_NTKConfig list? I had another developer work
>>on
>> this. I'll need to find out if he did.
>>
>> Thanks,
>>
>> Brad
>>
>> On 4/19/17, 5:34 PM, "Active4D-dev on behalf of Aparajita Fishman"
>> <[hidden email] on behalf of
>> [hidden email]> wrote:
>>
>>> Just to be clear, the Mac Mini is a client that is the Active4D web
>>> server?
>>>
>>> Have you turned on Active4D debug logging and looked for anything
>>> suspicious in the log?
>>>
>>> When you say that many A4D_HTTPD_RequestHandler processes were on the
>>> server, you mean the Mac Mini or the Windows server?
>>>
>>> Did you update the A4D_NTKConfig list?
>>>
>>> All the best,
>>>
>>> - Aparajita
>>>
>>>> On Apr 19, 2017, at 11:30 AM, Perkins, Bradley D <[hidden email]>
>>>> wrote:
>>>>
>>>> I noticed something else:
>>>>
>>>> On the production server the A4D_HouseKeeper process was of type "4D
>>>> Client Database Process" yet on development it was type "4D Client
>>>> Process".
>>>> When I relaunched the production server there is no A4D_HouseKeeper at
>>>> all.
>>>>
>>>> Brad
>>>>
>>>> From: Brad Perkins <[hidden email]<mailto:[hidden email]>>
>>>> Date: Wednesday, April 19, 2017 at 12:16 PM
>>>> To:
>>>>
>>>>"[hidden email]<mailto:[hidden email]
>>>>>"
>>>>
>>>>
>>>><[hidden email]<mailto:[hidden email]
>>>>>>
>>>> Subject: Active4D Slowdown then Stall (weird
>>>>A4D_HTTPD_RequestHandler_*
>>>> behavior)
>>>>
>>>> We upgraded a long running A4D-based system late last week and are
>>>> noticing that Active4D will eventually become unresponsive. We can't
>>>> serve for more than a day and web apps have been unresponsive the last
>>>> two mornings. The old system would run for a long time without fail.
>>>>
>>>> We went from
>>>> 4D 14.2 to 4D15.4
>>>> Active4D running on OS 10.11.latest to OS 10.12.4 (Sierra)
>>>> Active4D is behind Apache. The Mac is a "client". Server runs on
>>>> Windows.
>>>> Replaced a 6+ year old Xserve with a Mac Mini (16 GB RAM)
>>>> Upgraded to latest Active4D and latest NTK w/ necessary changes to
>>>> Shell Code.
>>>>
>>>> Admittedly a lot of changes, but this update was mandated by
>>>> institutional cybersecurity group (Sierra was required). Sierra would
>>>> not install on the Xserves. 15.4 is the only version close to
>>>>certified
>>>> for Sierra.
>>>>
>>>> When this occurs I find:
>>>> (a) Apache can still serve static content fine.
>>>> (b) 4D is still running and responsive to user input.
>>>> (c) I can see many A4D_HTTPD_RequestHandler_* processes on the server.
>>>> Many are of type "4D Client Database Process" with num "0" and State
>>>> "Running". That doesn't seem right to me.
>>>> I'm currently observing both our production and development servers
>>>>now
>>>> and neither show any "4D Client Database Process" processes. I can
>>>>force
>>>> the creation of new processes by using ApacheBench to make many
>>>> concurrent calls. All of the new process are client processes.
>>>>
>>>> Can anyone suggest how those  "4D Client Database Process" are being
>>>> created?
>>>>
>>>> Thanks,
>>>>
>>>> Brad Perkins
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Active4D-dev mailing list
>>>> [hidden email]
>>>> http://list.aparajitaworld.com/listinfo/active4d-dev
>>>> Archives: http://active4d-nabble.aparajitaworld.com/
>>>
>>> _______________________________________________
>>> Active4D-dev mailing list
>>> [hidden email]
>>> http://list.aparajitaworld.com/listinfo/active4d-dev
>>> Archives: http://active4d-nabble.aparajitaworld.com/
>>
>> _______________________________________________
>> Active4D-dev mailing list
>> [hidden email]
>> http://list.aparajitaworld.com/listinfo/active4d-dev
>> Archives: http://active4d-nabble.aparajitaworld.com/
>
>
>_______________________________________________
>Active4D-dev mailing list
>[hidden email]
>http://list.aparajitaworld.com/listinfo/active4d-dev
>Archives: http://active4d-nabble.aparajitaworld.com/

_______________________________________________
Active4D-dev mailing list
[hidden email]
http://list.aparajitaworld.com/listinfo/active4d-dev
Archives: http://active4d-nabble.aparajitaworld.com/
Reply | Threaded
Open this post in threaded view
|

Re: Active4D Slowdown then Stall (weird A4D_HTTPD_RequestHandler_* behavior)

Aparajita Fishman
In reply to this post by Aparajita Fishman
I take that back about the request handler processes, they should be on the server if they access data.

I’m fairly confident in the ability of the new NTK server code to handle high volumes, I tried a test using ab (apache benchmark) with a concurrency level of 20, making 1000 requests to a client as the web server, and it was getting 123 requests per second.

All the best,

  - Aparajita

> On Apr 20, 2017, at 7:48 AM, Aparajita Fishman <[hidden email]> wrote:
>
> What volume of requests are you getting before it stalls?
>
> You shouldn’t see the A4D_HTTPD_RequestHandler processes on the Windows server at all. They should only be on the client.
>
> All the best,
>
>  - Aparajita
>
>> On Apr 20, 2017, at 7:09 AM, Perkins, Bradley D <[hidden email]> wrote:
>>
>> Yes, the Mac Mini is the client that is the Active4D web server. Initially
>> we were having 'sleep' problems caused by Apple's built in power saving
>> settings. I've resolved those.
>>
>> I just arrived at work and can see that problems started ~4 am local time.
>> It is still down. I do have debug logging on and am going to look at those
>> shortly. I'll send you anything notable. I also want to see if I can
>> correlate any problems with incoming HTTPD requests from the Apache logs.
>>
>> The many A4D_HTTPD_RequestHandler processes were observed on the Windows
>> server. With the way our web client user interface is programmed it is
>> difficult to get to the Runtime environment but I'm going to try that too.
>>
>> re: Did you update the A4D_NTKConfig list? I had another developer work on
>> this. I'll need to find out if he did.
>>
>> Thanks,
>>
>> Brad
>>
>> On 4/19/17, 5:34 PM, "Active4D-dev on behalf of Aparajita Fishman"
>> <[hidden email] on behalf of
>> [hidden email]> wrote:
>>
>>> Just to be clear, the Mac Mini is a client that is the Active4D web
>>> server?
>>>
>>> Have you turned on Active4D debug logging and looked for anything
>>> suspicious in the log?
>>>
>>> When you say that many A4D_HTTPD_RequestHandler processes were on the
>>> server, you mean the Mac Mini or the Windows server?
>>>
>>> Did you update the A4D_NTKConfig list?
>>>
>>> All the best,
>>>
>>> - Aparajita
>>>
>>>> On Apr 19, 2017, at 11:30 AM, Perkins, Bradley D <[hidden email]>
>>>> wrote:
>>>>
>>>> I noticed something else:
>>>>
>>>> On the production server the A4D_HouseKeeper process was of type "4D
>>>> Client Database Process" yet on development it was type "4D Client
>>>> Process".
>>>> When I relaunched the production server there is no A4D_HouseKeeper at
>>>> all.
>>>>
>>>> Brad
>>>>
>>>> From: Brad Perkins <[hidden email]<mailto:[hidden email]>>
>>>> Date: Wednesday, April 19, 2017 at 12:16 PM
>>>> To:
>>>> "[hidden email]<mailto:[hidden email]>"
>>>>
>>>> <[hidden email]<mailto:[hidden email]>>
>>>> Subject: Active4D Slowdown then Stall (weird A4D_HTTPD_RequestHandler_*
>>>> behavior)
>>>>
>>>> We upgraded a long running A4D-based system late last week and are
>>>> noticing that Active4D will eventually become unresponsive. We can't
>>>> serve for more than a day and web apps have been unresponsive the last
>>>> two mornings. The old system would run for a long time without fail.
>>>>
>>>> We went from
>>>> 4D 14.2 to 4D15.4
>>>> Active4D running on OS 10.11.latest to OS 10.12.4 (Sierra)
>>>> Active4D is behind Apache. The Mac is a "client". Server runs on
>>>> Windows.
>>>> Replaced a 6+ year old Xserve with a Mac Mini (16 GB RAM)
>>>> Upgraded to latest Active4D and latest NTK w/ necessary changes to
>>>> Shell Code.
>>>>
>>>> Admittedly a lot of changes, but this update was mandated by
>>>> institutional cybersecurity group (Sierra was required). Sierra would
>>>> not install on the Xserves. 15.4 is the only version close to certified
>>>> for Sierra.
>>>>
>>>> When this occurs I find:
>>>> (a) Apache can still serve static content fine.
>>>> (b) 4D is still running and responsive to user input.
>>>> (c) I can see many A4D_HTTPD_RequestHandler_* processes on the server.
>>>> Many are of type "4D Client Database Process" with num "0" and State
>>>> "Running". That doesn't seem right to me.
>>>> I'm currently observing both our production and development servers now
>>>> and neither show any "4D Client Database Process" processes. I can force
>>>> the creation of new processes by using ApacheBench to make many
>>>> concurrent calls. All of the new process are client processes.
>>>>
>>>> Can anyone suggest how those  "4D Client Database Process" are being
>>>> created?
>>>>
>>>> Thanks,
>>>>
>>>> Brad Perkins
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Active4D-dev mailing list
>>>> [hidden email]
>>>> http://list.aparajitaworld.com/listinfo/active4d-dev
>>>> Archives: http://active4d-nabble.aparajitaworld.com/
>>>
>>> _______________________________________________
>>> Active4D-dev mailing list
>>> [hidden email]
>>> http://list.aparajitaworld.com/listinfo/active4d-dev
>>> Archives: http://active4d-nabble.aparajitaworld.com/
>>
>> _______________________________________________
>> Active4D-dev mailing list
>> [hidden email]
>> http://list.aparajitaworld.com/listinfo/active4d-dev
>> Archives: http://active4d-nabble.aparajitaworld.com/
>
>
> _______________________________________________
> Active4D-dev mailing list
> [hidden email]
> http://list.aparajitaworld.com/listinfo/active4d-dev
> Archives: http://active4d-nabble.aparajitaworld.com/


_______________________________________________
Active4D-dev mailing list
[hidden email]
http://list.aparajitaworld.com/listinfo/active4d-dev
Archives: http://active4d-nabble.aparajitaworld.com/
Reply | Threaded
Open this post in threaded view
|

Re: Active4D Slowdown then Stall (weird A4D_HTTPD_RequestHandler_* behavior)

Perkins, Bradley D
In our case the request handlers typically access data.

I've used ab too with fairly high numbers and not seen problems. In terms
of higher volume real world access we have allowed our program's googlebox
to index some content that is served by A4D. They do this frequently but
at a rate of a request every few seconds (logs show ~16400 requests
yesterday). I also have external monitors that make request every few
minutes for a simple status page.

Thanks,

Brad

On 4/20/17, 9:01 AM, "Active4D-dev on behalf of Aparajita Fishman"
<[hidden email] on behalf of
[hidden email]> wrote:

>I take that back about the request handler processes, they should be on
>the server if they access data.
>
>I’m fairly confident in the ability of the new NTK server code to handle
>high volumes, I tried a test using ab (apache benchmark) with a
>concurrency level of 20, making 1000 requests to a client as the web
>server, and it was getting 123 requests per second.
>
>All the best,
>
>  - Aparajita
>
>> On Apr 20, 2017, at 7:48 AM, Aparajita Fishman
>><[hidden email]> wrote:
>>
>> What volume of requests are you getting before it stalls?
>>
>> You shouldn’t see the A4D_HTTPD_RequestHandler processes on the Windows
>>server at all. They should only be on the client.
>>
>> All the best,
>>
>>  - Aparajita
>>
>>> On Apr 20, 2017, at 7:09 AM, Perkins, Bradley D <[hidden email]>
>>>wrote:
>>>
>>> Yes, the Mac Mini is the client that is the Active4D web server.
>>>Initially
>>> we were having 'sleep' problems caused by Apple's built in power saving
>>> settings. I've resolved those.
>>>
>>> I just arrived at work and can see that problems started ~4 am local
>>>time.
>>> It is still down. I do have debug logging on and am going to look at
>>>those
>>> shortly. I'll send you anything notable. I also want to see if I can
>>> correlate any problems with incoming HTTPD requests from the Apache
>>>logs.
>>>
>>> The many A4D_HTTPD_RequestHandler processes were observed on the
>>>Windows
>>> server. With the way our web client user interface is programmed it is
>>> difficult to get to the Runtime environment but I'm going to try that
>>>too.
>>>
>>> re: Did you update the A4D_NTKConfig list? I had another developer
>>>work on
>>> this. I'll need to find out if he did.
>>>
>>> Thanks,
>>>
>>> Brad
>>>
>>> On 4/19/17, 5:34 PM, "Active4D-dev on behalf of Aparajita Fishman"
>>> <[hidden email] on behalf of
>>> [hidden email]> wrote:
>>>
>>>> Just to be clear, the Mac Mini is a client that is the Active4D web
>>>> server?
>>>>
>>>> Have you turned on Active4D debug logging and looked for anything
>>>> suspicious in the log?
>>>>
>>>> When you say that many A4D_HTTPD_RequestHandler processes were on the
>>>> server, you mean the Mac Mini or the Windows server?
>>>>
>>>> Did you update the A4D_NTKConfig list?
>>>>
>>>> All the best,
>>>>
>>>> - Aparajita
>>>>
>>>>> On Apr 19, 2017, at 11:30 AM, Perkins, Bradley D <[hidden email]>
>>>>> wrote:
>>>>>
>>>>> I noticed something else:
>>>>>
>>>>> On the production server the A4D_HouseKeeper process was of type "4D
>>>>> Client Database Process" yet on development it was type "4D Client
>>>>> Process".
>>>>> When I relaunched the production server there is no A4D_HouseKeeper
>>>>>at
>>>>> all.
>>>>>
>>>>> Brad
>>>>>
>>>>> From: Brad Perkins <[hidden email]<mailto:[hidden email]>>
>>>>> Date: Wednesday, April 19, 2017 at 12:16 PM
>>>>> To:
>>>>>
>>>>>"[hidden email]<mailto:[hidden email]
>>>>>m>"
>>>>>
>>>>>
>>>>><[hidden email]<mailto:[hidden email]
>>>>>m>>
>>>>> Subject: Active4D Slowdown then Stall (weird
>>>>>A4D_HTTPD_RequestHandler_*
>>>>> behavior)
>>>>>
>>>>> We upgraded a long running A4D-based system late last week and are
>>>>> noticing that Active4D will eventually become unresponsive. We can't
>>>>> serve for more than a day and web apps have been unresponsive the
>>>>>last
>>>>> two mornings. The old system would run for a long time without fail.
>>>>>
>>>>> We went from
>>>>> 4D 14.2 to 4D15.4
>>>>> Active4D running on OS 10.11.latest to OS 10.12.4 (Sierra)
>>>>> Active4D is behind Apache. The Mac is a "client". Server runs on
>>>>> Windows.
>>>>> Replaced a 6+ year old Xserve with a Mac Mini (16 GB RAM)
>>>>> Upgraded to latest Active4D and latest NTK w/ necessary changes to
>>>>> Shell Code.
>>>>>
>>>>> Admittedly a lot of changes, but this update was mandated by
>>>>> institutional cybersecurity group (Sierra was required). Sierra would
>>>>> not install on the Xserves. 15.4 is the only version close to
>>>>>certified
>>>>> for Sierra.
>>>>>
>>>>> When this occurs I find:
>>>>> (a) Apache can still serve static content fine.
>>>>> (b) 4D is still running and responsive to user input.
>>>>> (c) I can see many A4D_HTTPD_RequestHandler_* processes on the
>>>>>server.
>>>>> Many are of type "4D Client Database Process" with num "0" and State
>>>>> "Running". That doesn't seem right to me.
>>>>> I'm currently observing both our production and development servers
>>>>>now
>>>>> and neither show any "4D Client Database Process" processes. I can
>>>>>force
>>>>> the creation of new processes by using ApacheBench to make many
>>>>> concurrent calls. All of the new process are client processes.
>>>>>
>>>>> Can anyone suggest how those  "4D Client Database Process" are being
>>>>> created?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Brad Perkins
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Active4D-dev mailing list
>>>>> [hidden email]
>>>>> http://list.aparajitaworld.com/listinfo/active4d-dev
>>>>> Archives: http://active4d-nabble.aparajitaworld.com/
>>>>
>>>> _______________________________________________
>>>> Active4D-dev mailing list
>>>> [hidden email]
>>>> http://list.aparajitaworld.com/listinfo/active4d-dev
>>>> Archives: http://active4d-nabble.aparajitaworld.com/
>>>
>>> _______________________________________________
>>> Active4D-dev mailing list
>>> [hidden email]
>>> http://list.aparajitaworld.com/listinfo/active4d-dev
>>> Archives: http://active4d-nabble.aparajitaworld.com/
>>
>>
>> _______________________________________________
>> Active4D-dev mailing list
>> [hidden email]
>> http://list.aparajitaworld.com/listinfo/active4d-dev
>> Archives: http://active4d-nabble.aparajitaworld.com/
>
>
>_______________________________________________
>Active4D-dev mailing list
>[hidden email]
>http://list.aparajitaworld.com/listinfo/active4d-dev
>Archives: http://active4d-nabble.aparajitaworld.com/

_______________________________________________
Active4D-dev mailing list
[hidden email]
http://list.aparajitaworld.com/listinfo/active4d-dev
Archives: http://active4d-nabble.aparajitaworld.com/
Reply | Threaded
Open this post in threaded view
|

Re: Active4D Slowdown then Stall (weird A4D_HTTPD_RequestHandler_* behavior)

Aparajita Fishman
You have to tweak your config in the A4D_NTKConfig list.

The old server code had a broken implementation of keep-alive which never actually waited for subsequent requests on the same thread. So each thread served a single request and immediately became available.

The new implementation waits for the number of seconds specified in KeepAliveTimeout, which defaults to 5, before it becomes available to serve another request. So once you reach MaxThreads, each new request could potentially wait up to 5 seconds to be served. Thus if you then continue to get new requests at a rate > KeepAliveTimeout per second, the server slows to a crawl.

Note that this is not a deficiency in the design of the server; apache also waits 5 seconds per thread by default and thus has the same problem, but its limit is much higher. Once it reaches that limit it slows to a crawl, and that’s what a denial of service attack intentionally tries to do.

So the solution is to increase MaxThreads to a much higher number. This can be arbitrarily large as long as you have lots of memory available. You could tweak ThreadStack accordingly. Currently there is no code to reclaim threads after a period of inactivity.

All the best,

  - Aparajita

> Since 4D continues to run fine I suspected that we are getting into a situation where the Active4D/NTK server exhausts all of its threads. In other words it has started the max thread count, and those threads exist, but are in a state where they are not available. What I don't know is why that would happen.
>
> By running some very heavy tests with Apache Bench, I could repeatedly see the allocation of new threads up until the point where <>A4D_ThreadPool_ThreadIDs) = <>A4D_ThreadPool_MaxThreads.
> At that point our system become unresponsive. While these heavy test were running our Active4D pages were responsive for me. The logs also didn't show any errors.
>
> In my case I could always kill the server if I entered an ab concurrency value that was higher than the <>A4D_ThreadPool_MaxThreads which is 27 in our case. 30 would kill it.
>
> Do you have any ideas?


_______________________________________________
Active4D-dev mailing list
[hidden email]
http://list.aparajitaworld.com/listinfo/active4d-dev
Archives: http://active4d-nabble.aparajitaworld.com/
Reply | Threaded
Open this post in threaded view
|

Re: Active4D Slowdown then Stall (weird A4D_HTTPD_RequestHandler_* behavior)

Peter Gutbrod
In reply to this post by Perkins, Bradley D
Hi Brad,

I see something similar since I've upgraded 4D-Server to V16 and NTK/A4D
to the newest version.

Before the server could run für month without any issue. After the
upgrade the server slows down within 1-2 days up the point, where each
request can take 10 seconds or even more. According to the 4D admin
panel there are cpu or network intensive tasks running. Server is almost
idle, tons af free memory for 4D. Nothing suspicious in Windows
performance monitor. Rebooting 4D and the slow-down is gone -  for the
next couple of hours, then it slowly reappears.

I realized the slowdown in a JSON API. First I thought it is something
within my A4D code, so I've calculated the time my A4D code needs to
run. But I found the slowdown happens while receiving the request. Once
A4D starts executing the code, everything runs fast, including sending
the response.

I've asked Rob from the pluggers, whether he had heard about slowdows
with the new version of NTK but he wasn't aware of any problems.

In a period, where I wasn't able to restart 4D server periodically,
slowdown got so extreme, that 4D clients were unable to login due to
timeouts. In that state the only way to restart the 4D server was with a
force quit. That convinced me, that the culprit is within 4D V16 and not
NTK or A4D and I should wait for 4D 16.1 and until then restart 4D
Server periodically. That is why I never reported the problem here.

As you have a similar problem but with 4D15.4 makes me wondering,
whether I'm wrong and the lastest versions of NTK and A4D might at least
contribute to the problem.

Hope you find the answer and share it with me.

Greetings

Peter Gutbrod
_______________________________________________
Active4D-dev mailing list
[hidden email]
http://list.aparajitaworld.com/listinfo/active4d-dev
Archives: http://active4d-nabble.aparajitaworld.com/
Reply | Threaded
Open this post in threaded view
|

Re: Active4D Slowdown then Stall (weird A4D_HTTPD_RequestHandler_* behavior)

Peter Gutbrod


Peter Gutbrod schrieb:
> According to the 4D admin
> panel there are cpu or network intensive tasks running
 Should be:

there are NO cpu or network intensive tasks running ;-)


_______________________________________________
Active4D-dev mailing list
[hidden email]
http://list.aparajitaworld.com/listinfo/active4d-dev
Archives: http://active4d-nabble.aparajitaworld.com/
Reply | Threaded
Open this post in threaded view
|

Re: Active4D Slowdown then Stall (weird A4D_HTTPD_RequestHandler_* behavior)

Aparajita Fishman
In reply to this post by Peter Gutbrod
Increase MaxThreads in the A4D_NTKConfig list. I am increasing this to 100 by default in the next release, and I am also adding code to reclaim unused threads after a configurable delay.

All the best,

  - Aparajita

> On Apr 24, 2017, at 4:51 AM, Peter Gutbrod <[hidden email]> wrote:
>
> Hi Brad,
>
> I see something similar since I've upgraded 4D-Server to V16 and NTK/A4D
> to the newest version.
>
> Before the server could run für month without any issue. After the
> upgrade the server slows down within 1-2 days up the point, where each
> request can take 10 seconds or even more. According to the 4D admin
> panel there are cpu or network intensive tasks running. Server is almost
> idle, tons af free memory for 4D. Nothing suspicious in Windows
> performance monitor. Rebooting 4D and the slow-down is gone -  for the
> next couple of hours, then it slowly reappears.
>
> I realized the slowdown in a JSON API. First I thought it is something
> within my A4D code, so I've calculated the time my A4D code needs to
> run. But I found the slowdown happens while receiving the request. Once
> A4D starts executing the code, everything runs fast, including sending
> the response.
>
> I've asked Rob from the pluggers, whether he had heard about slowdows
> with the new version of NTK but he wasn't aware of any problems.
>
> In a period, where I wasn't able to restart 4D server periodically,
> slowdown got so extreme, that 4D clients were unable to login due to
> timeouts. In that state the only way to restart the 4D server was with a
> force quit. That convinced me, that the culprit is within 4D V16 and not
> NTK or A4D and I should wait for 4D 16.1 and until then restart 4D
> Server periodically. That is why I never reported the problem here.
>
> As you have a similar problem but with 4D15.4 makes me wondering,
> whether I'm wrong and the lastest versions of NTK and A4D might at least
> contribute to the problem.
>
> Hope you find the answer and share it with me.
>
> Greetings
>
> Peter Gutbrod
> _______________________________________________
> Active4D-dev mailing list
> [hidden email]
> http://list.aparajitaworld.com/listinfo/active4d-dev
> Archives: http://active4d-nabble.aparajitaworld.com/


_______________________________________________
Active4D-dev mailing list
[hidden email]
http://list.aparajitaworld.com/listinfo/active4d-dev
Archives: http://active4d-nabble.aparajitaworld.com/
Reply | Threaded
Open this post in threaded view
|

Re: Active4D Slowdown then Stall (weird A4D_HTTPD_RequestHandler_* behavior)

Perkins, Bradley D
Peter and Aparajita,

Thank you both for your responses. I have been testing our development
system with a higher MaxThreads this morning. I haven't been able to
exhaust all of the threads. I was able to crash 4D through. I'll send
Aparajita the crash log.

Peter, your observations are very similar to what we've seen. Under v14 --
and even with the broken keep alive feature -- we could run for a long
time with no problems. Reboots were usually because I had to reboot the
Windows server to install MS Security Patches.

Our production system sees a fairly light load and would normally never
see the loads that I'm placing on it with ApacheBench unless IT is running
a cyber scan. I'm letting it run as is with the earlier configuration to
see if the problem is because we've exhausted the thread pool. We haven't
allocated an additional thread since I restarted on Saturday. Our
ThreadStack size was already set to 256 (docs show 128 as default).
As Peter notes, It could be that the problem lies elsewhere such as in in
4D itself.

We upgraded hardware, OS, 4D, NTK, and Active4D. That is a lot of change.

-- Brad

On 4/24/17, 9:09 AM, "Active4D-dev on behalf of Aparajita Fishman"
<[hidden email] on behalf of
[hidden email]> wrote:

>Increase MaxThreads in the A4D_NTKConfig list. I am increasing this to
>100 by default in the next release, and I am also adding code to reclaim
>unused threads after a configurable delay.
>
>All the best,
>
>  - Aparajita
>
>> On Apr 24, 2017, at 4:51 AM, Peter Gutbrod <[hidden email]> wrote:
>>
>> Hi Brad,
>>
>> I see something similar since I've upgraded 4D-Server to V16 and NTK/A4D
>> to the newest version.
>>
>> Before the server could run für month without any issue. After the
>> upgrade the server slows down within 1-2 days up the point, where each
>> request can take 10 seconds or even more. According to the 4D admin
>> panel there are cpu or network intensive tasks running. Server is almost
>> idle, tons af free memory for 4D. Nothing suspicious in Windows
>> performance monitor. Rebooting 4D and the slow-down is gone -  for the
>> next couple of hours, then it slowly reappears.
>>
>> I realized the slowdown in a JSON API. First I thought it is something
>> within my A4D code, so I've calculated the time my A4D code needs to
>> run. But I found the slowdown happens while receiving the request. Once
>> A4D starts executing the code, everything runs fast, including sending
>> the response.
>>
>> I've asked Rob from the pluggers, whether he had heard about slowdows
>> with the new version of NTK but he wasn't aware of any problems.
>>
>> In a period, where I wasn't able to restart 4D server periodically,
>> slowdown got so extreme, that 4D clients were unable to login due to
>> timeouts. In that state the only way to restart the 4D server was with a
>> force quit. That convinced me, that the culprit is within 4D V16 and not
>> NTK or A4D and I should wait for 4D 16.1 and until then restart 4D
>> Server periodically. That is why I never reported the problem here.
>>
>> As you have a similar problem but with 4D15.4 makes me wondering,
>> whether I'm wrong and the lastest versions of NTK and A4D might at least
>> contribute to the problem.
>>
>> Hope you find the answer and share it with me.
>>
>> Greetings
>>
>> Peter Gutbrod
>> _______________________________________________
>> Active4D-dev mailing list
>> [hidden email]
>> http://list.aparajitaworld.com/listinfo/active4d-dev
>> Archives: http://active4d-nabble.aparajitaworld.com/
>
>
>_______________________________________________
>Active4D-dev mailing list
>[hidden email]
>http://list.aparajitaworld.com/listinfo/active4d-dev
>Archives: http://active4d-nabble.aparajitaworld.com/

_______________________________________________
Active4D-dev mailing list
[hidden email]
http://list.aparajitaworld.com/listinfo/active4d-dev
Archives: http://active4d-nabble.aparajitaworld.com/
Reply | Threaded
Open this post in threaded view
|

Re: Active4D Slowdown then Stall (weird A4D_HTTPD_RequestHandler_* behavior)

Aparajita Fishman
It’s entirely possible there is some flaw in the new server code. The main change I made to A4D_HTTPD_RequestHandler was the correct handling of keep-alive connections. If you find out any more details, please post them here.

All the best,

  - Aparajita

> On Apr 24, 2017, at 8:43 AM, Perkins, Bradley D <[hidden email]> wrote:
>
> Peter and Aparajita,
>
> Thank you both for your responses. I have been testing our development
> system with a higher MaxThreads this morning. I haven't been able to
> exhaust all of the threads. I was able to crash 4D through. I'll send
> Aparajita the crash log.
>
> Peter, your observations are very similar to what we've seen. Under v14 --
> and even with the broken keep alive feature -- we could run for a long
> time with no problems. Reboots were usually because I had to reboot the
> Windows server to install MS Security Patches.
>
> Our production system sees a fairly light load and would normally never
> see the loads that I'm placing on it with ApacheBench unless IT is running
> a cyber scan. I'm letting it run as is with the earlier configuration to
> see if the problem is because we've exhausted the thread pool. We haven't
> allocated an additional thread since I restarted on Saturday. Our
> ThreadStack size was already set to 256 (docs show 128 as default).
> As Peter notes, It could be that the problem lies elsewhere such as in in
> 4D itself.
>
> We upgraded hardware, OS, 4D, NTK, and Active4D. That is a lot of change.
>
> -- Brad
>
> On 4/24/17, 9:09 AM, "Active4D-dev on behalf of Aparajita Fishman"
> <[hidden email] on behalf of
> [hidden email]> wrote:
>
>> Increase MaxThreads in the A4D_NTKConfig list. I am increasing this to
>> 100 by default in the next release, and I am also adding code to reclaim
>> unused threads after a configurable delay.
>>
>> All the best,
>>
>> - Aparajita
>>
>>> On Apr 24, 2017, at 4:51 AM, Peter Gutbrod <[hidden email]> wrote:
>>>
>>> Hi Brad,
>>>
>>> I see something similar since I've upgraded 4D-Server to V16 and NTK/A4D
>>> to the newest version.
>>>
>>> Before the server could run für month without any issue. After the
>>> upgrade the server slows down within 1-2 days up the point, where each
>>> request can take 10 seconds or even more. According to the 4D admin
>>> panel there are cpu or network intensive tasks running. Server is almost
>>> idle, tons af free memory for 4D. Nothing suspicious in Windows
>>> performance monitor. Rebooting 4D and the slow-down is gone -  for the
>>> next couple of hours, then it slowly reappears.
>>>
>>> I realized the slowdown in a JSON API. First I thought it is something
>>> within my A4D code, so I've calculated the time my A4D code needs to
>>> run. But I found the slowdown happens while receiving the request. Once
>>> A4D starts executing the code, everything runs fast, including sending
>>> the response.
>>>
>>> I've asked Rob from the pluggers, whether he had heard about slowdows
>>> with the new version of NTK but he wasn't aware of any problems.
>>>
>>> In a period, where I wasn't able to restart 4D server periodically,
>>> slowdown got so extreme, that 4D clients were unable to login due to
>>> timeouts. In that state the only way to restart the 4D server was with a
>>> force quit. That convinced me, that the culprit is within 4D V16 and not
>>> NTK or A4D and I should wait for 4D 16.1 and until then restart 4D
>>> Server periodically. That is why I never reported the problem here.
>>>
>>> As you have a similar problem but with 4D15.4 makes me wondering,
>>> whether I'm wrong and the lastest versions of NTK and A4D might at least
>>> contribute to the problem.
>>>
>>> Hope you find the answer and share it with me.
>>>
>>> Greetings
>>>
>>> Peter Gutbrod
>>> _______________________________________________
>>> Active4D-dev mailing list
>>> [hidden email]
>>> http://list.aparajitaworld.com/listinfo/active4d-dev
>>> Archives: http://active4d-nabble.aparajitaworld.com/
>>
>>
>> _______________________________________________
>> Active4D-dev mailing list
>> [hidden email]
>> http://list.aparajitaworld.com/listinfo/active4d-dev
>> Archives: http://active4d-nabble.aparajitaworld.com/
>
> _______________________________________________
> Active4D-dev mailing list
> [hidden email]
> http://list.aparajitaworld.com/listinfo/active4d-dev
> Archives: http://active4d-nabble.aparajitaworld.com/


_______________________________________________
Active4D-dev mailing list
[hidden email]
http://list.aparajitaworld.com/listinfo/active4d-dev
Archives: http://active4d-nabble.aparajitaworld.com/
Reply | Threaded
Open this post in threaded view
|

Re: Active4D Slowdown then Stall (weird A4D_HTTPD_RequestHandler_* behavior)

John Bellos
I forget what version of 4D you said you came from, but if it was v13 or lower, try your setup with the new network layer OFF in 4D server. We've definitely seen some stability issues with the new network layer ON, using v15.4.

Note that if you're using Mac 64-Bit, the new network layer is forced on and you can't turn it off. But on Windows 64 bit server, you can turn this setting off.

________________________________________
From: Active4D-dev [[hidden email]] on behalf of Aparajita Fishman [[hidden email]]
Sent: Monday, April 24, 2017 4:29 PM
To: Active4D Developer Discussion List
Subject: Re: [Active4d-dev] Active4D Slowdown then Stall (weird A4D_HTTPD_RequestHandler_* behavior)

It’s entirely possible there is some flaw in the new server code. The main change I made to A4D_HTTPD_RequestHandler was the correct handling of keep-alive connections. If you find out any more details, please post them here.

All the best,

  - Aparajita

> On Apr 24, 2017, at 8:43 AM, Perkins, Bradley D <[hidden email]> wrote:
>
> Peter and Aparajita,
>
> Thank you both for your responses. I have been testing our development
> system with a higher MaxThreads this morning. I haven't been able to
> exhaust all of the threads. I was able to crash 4D through. I'll send
> Aparajita the crash log.
>
> Peter, your observations are very similar to what we've seen. Under v14 --
> and even with the broken keep alive feature -- we could run for a long
> time with no problems. Reboots were usually because I had to reboot the
> Windows server to install MS Security Patches.
>
> Our production system sees a fairly light load and would normally never
> see the loads that I'm placing on it with ApacheBench unless IT is running
> a cyber scan. I'm letting it run as is with the earlier configuration to
> see if the problem is because we've exhausted the thread pool. We haven't
> allocated an additional thread since I restarted on Saturday. Our
> ThreadStack size was already set to 256 (docs show 128 as default).
> As Peter notes, It could be that the problem lies elsewhere such as in in
> 4D itself.
>
> We upgraded hardware, OS, 4D, NTK, and Active4D. That is a lot of change.
>
> -- Brad
>
> On 4/24/17, 9:09 AM, "Active4D-dev on behalf of Aparajita Fishman"
> <[hidden email] on behalf of
> [hidden email]> wrote:
>
>> Increase MaxThreads in the A4D_NTKConfig list. I am increasing this to
>> 100 by default in the next release, and I am also adding code to reclaim
>> unused threads after a configurable delay.
>>
>> All the best,
>>
>> - Aparajita
>>
>>> On Apr 24, 2017, at 4:51 AM, Peter Gutbrod <[hidden email]> wrote:
>>>
>>> Hi Brad,
>>>
>>> I see something similar since I've upgraded 4D-Server to V16 and NTK/A4D
>>> to the newest version.
>>>
>>> Before the server could run für month without any issue. After the
>>> upgrade the server slows down within 1-2 days up the point, where each
>>> request can take 10 seconds or even more. According to the 4D admin
>>> panel there are cpu or network intensive tasks running. Server is almost
>>> idle, tons af free memory for 4D. Nothing suspicious in Windows
>>> performance monitor. Rebooting 4D and the slow-down is gone -  for the
>>> next couple of hours, then it slowly reappears.
>>>
>>> I realized the slowdown in a JSON API. First I thought it is something
>>> within my A4D code, so I've calculated the time my A4D code needs to
>>> run. But I found the slowdown happens while receiving the request. Once
>>> A4D starts executing the code, everything runs fast, including sending
>>> the response.
>>>
>>> I've asked Rob from the pluggers, whether he had heard about slowdows
>>> with the new version of NTK but he wasn't aware of any problems.
>>>
>>> In a period, where I wasn't able to restart 4D server periodically,
>>> slowdown got so extreme, that 4D clients were unable to login due to
>>> timeouts. In that state the only way to restart the 4D server was with a
>>> force quit. That convinced me, that the culprit is within 4D V16 and not
>>> NTK or A4D and I should wait for 4D 16.1 and until then restart 4D
>>> Server periodically. That is why I never reported the problem here.
>>>
>>> As you have a similar problem but with 4D15.4 makes me wondering,
>>> whether I'm wrong and the lastest versions of NTK and A4D might at least
>>> contribute to the problem.
>>>
>>> Hope you find the answer and share it with me.
>>>
>>> Greetings
>>>
>>> Peter Gutbrod
>>> _______________________________________________
>>> Active4D-dev mailing list
>>> [hidden email]
>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__list.aparajitaworld.com_listinfo_active4d-2Ddev&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8UQmO8P5M3-rwSBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wobVcHPsDls&s=TJmubZ_4H7iiYRramtlMy84fx1lyLkuOJfA391x78cs&e=
>>> Archives: https://urldefense.proofpoint.com/v2/url?u=http-3A__active4d-2Dnabble.aparajitaworld.com_&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8UQmO8P5M3-rwSBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wobVcHPsDls&s=VJzLTdVZBbjHRZH1usEIxPXoRTc4FrJF5QANN_xTdRk&e=
>>
>>
>> _______________________________________________
>> Active4D-dev mailing list
>> [hidden email]
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__list.aparajitaworld.com_listinfo_active4d-2Ddev&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8UQmO8P5M3-rwSBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wobVcHPsDls&s=TJmubZ_4H7iiYRramtlMy84fx1lyLkuOJfA391x78cs&e=
>> Archives: https://urldefense.proofpoint.com/v2/url?u=http-3A__active4d-2Dnabble.aparajitaworld.com_&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8UQmO8P5M3-rwSBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wobVcHPsDls&s=VJzLTdVZBbjHRZH1usEIxPXoRTc4FrJF5QANN_xTdRk&e=
>
> _______________________________________________
> Active4D-dev mailing list
> [hidden email]
> https://urldefense.proofpoint.com/v2/url?u=http-3A__list.aparajitaworld.com_listinfo_active4d-2Ddev&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8UQmO8P5M3-rwSBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wobVcHPsDls&s=TJmubZ_4H7iiYRramtlMy84fx1lyLkuOJfA391x78cs&e=
> Archives: https://urldefense.proofpoint.com/v2/url?u=http-3A__active4d-2Dnabble.aparajitaworld.com_&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8UQmO8P5M3-rwSBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wobVcHPsDls&s=VJzLTdVZBbjHRZH1usEIxPXoRTc4FrJF5QANN_xTdRk&e=


_______________________________________________
Active4D-dev mailing list
[hidden email]
https://urldefense.proofpoint.com/v2/url?u=http-3A__list.aparajitaworld.com_listinfo_active4d-2Ddev&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8UQmO8P5M3-rwSBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wobVcHPsDls&s=TJmubZ_4H7iiYRramtlMy84fx1lyLkuOJfA391x78cs&e=
Archives: https://urldefense.proofpoint.com/v2/url?u=http-3A__active4d-2Dnabble.aparajitaworld.com_&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8UQmO8P5M3-rwSBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wobVcHPsDls&s=VJzLTdVZBbjHRZH1usEIxPXoRTc4FrJF5QANN_xTdRk&e=
_______________________________________________
Active4D-dev mailing list
[hidden email]
http://list.aparajitaworld.com/listinfo/active4d-dev
Archives: http://active4d-nabble.aparajitaworld.com/
Reply | Threaded
Open this post in threaded view
|

Re: Active4D Slowdown then Stall (weird A4D_HTTPD_RequestHandler_* behavior)

Perkins, Bradley D
Hi John,

We went from v14 to v15. We're also still running 32-bit server. I'm not
positive but I thought the new network layer was only available in 64-bit.
I will definitely check though. Thanks!

Brad

On 4/24/17, 2:31 PM, "Active4D-dev on behalf of Bellos, John"
<[hidden email] on behalf of
[hidden email]> wrote:

>I forget what version of 4D you said you came from, but if it was v13 or
>lower, try your setup with the new network layer OFF in 4D server. We've
>definitely seen some stability issues with the new network layer ON,
>using v15.4.
>
>Note that if you're using Mac 64-Bit, the new network layer is forced on
>and you can't turn it off. But on Windows 64 bit server, you can turn
>this setting off.
>
>________________________________________
>From: Active4D-dev [[hidden email]] on behalf of
>Aparajita Fishman [[hidden email]]
>Sent: Monday, April 24, 2017 4:29 PM
>To: Active4D Developer Discussion List
>Subject: Re: [Active4d-dev] Active4D Slowdown then Stall (weird
>A4D_HTTPD_RequestHandler_* behavior)
>
>It’s entirely possible there is some flaw in the new server code. The
>main change I made to A4D_HTTPD_RequestHandler was the correct handling
>of keep-alive connections. If you find out any more details, please post
>them here.
>
>All the best,
>
>  - Aparajita
>
>> On Apr 24, 2017, at 8:43 AM, Perkins, Bradley D <[hidden email]>
>>wrote:
>>
>> Peter and Aparajita,
>>
>> Thank you both for your responses. I have been testing our development
>> system with a higher MaxThreads this morning. I haven't been able to
>> exhaust all of the threads. I was able to crash 4D through. I'll send
>> Aparajita the crash log.
>>
>> Peter, your observations are very similar to what we've seen. Under v14
>>--
>> and even with the broken keep alive feature -- we could run for a long
>> time with no problems. Reboots were usually because I had to reboot the
>> Windows server to install MS Security Patches.
>>
>> Our production system sees a fairly light load and would normally never
>> see the loads that I'm placing on it with ApacheBench unless IT is
>>running
>> a cyber scan. I'm letting it run as is with the earlier configuration to
>> see if the problem is because we've exhausted the thread pool. We
>>haven't
>> allocated an additional thread since I restarted on Saturday. Our
>> ThreadStack size was already set to 256 (docs show 128 as default).
>> As Peter notes, It could be that the problem lies elsewhere such as in
>>in
>> 4D itself.
>>
>> We upgraded hardware, OS, 4D, NTK, and Active4D. That is a lot of
>>change.
>>
>> -- Brad
>>
>> On 4/24/17, 9:09 AM, "Active4D-dev on behalf of Aparajita Fishman"
>> <[hidden email] on behalf of
>> [hidden email]> wrote:
>>
>>> Increase MaxThreads in the A4D_NTKConfig list. I am increasing this to
>>> 100 by default in the next release, and I am also adding code to
>>>reclaim
>>> unused threads after a configurable delay.
>>>
>>> All the best,
>>>
>>> - Aparajita
>>>
>>>> On Apr 24, 2017, at 4:51 AM, Peter Gutbrod <[hidden email]>
>>>>wrote:
>>>>
>>>> Hi Brad,
>>>>
>>>> I see something similar since I've upgraded 4D-Server to V16 and
>>>>NTK/A4D
>>>> to the newest version.
>>>>
>>>> Before the server could run für month without any issue. After the
>>>> upgrade the server slows down within 1-2 days up the point, where each
>>>> request can take 10 seconds or even more. According to the 4D admin
>>>> panel there are cpu or network intensive tasks running. Server is
>>>>almost
>>>> idle, tons af free memory for 4D. Nothing suspicious in Windows
>>>> performance monitor. Rebooting 4D and the slow-down is gone -  for the
>>>> next couple of hours, then it slowly reappears.
>>>>
>>>> I realized the slowdown in a JSON API. First I thought it is something
>>>> within my A4D code, so I've calculated the time my A4D code needs to
>>>> run. But I found the slowdown happens while receiving the request.
>>>>Once
>>>> A4D starts executing the code, everything runs fast, including sending
>>>> the response.
>>>>
>>>> I've asked Rob from the pluggers, whether he had heard about slowdows
>>>> with the new version of NTK but he wasn't aware of any problems.
>>>>
>>>> In a period, where I wasn't able to restart 4D server periodically,
>>>> slowdown got so extreme, that 4D clients were unable to login due to
>>>> timeouts. In that state the only way to restart the 4D server was
>>>>with a
>>>> force quit. That convinced me, that the culprit is within 4D V16 and
>>>>not
>>>> NTK or A4D and I should wait for 4D 16.1 and until then restart 4D
>>>> Server periodically. That is why I never reported the problem here.
>>>>
>>>> As you have a similar problem but with 4D15.4 makes me wondering,
>>>> whether I'm wrong and the lastest versions of NTK and A4D might at
>>>>least
>>>> contribute to the problem.
>>>>
>>>> Hope you find the answer and share it with me.
>>>>
>>>> Greetings
>>>>
>>>> Peter Gutbrod
>>>> _______________________________________________
>>>> Active4D-dev mailing list
>>>> [hidden email]
>>>>
>>>>https://urldefense.proofpoint.com/v2/url?u=http-3A__list.aparajitaworld
>>>>.com_listinfo_active4d-2Ddev&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVR
>>>>q8UQmO8P5M3-rwSBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC
>>>>6wobVcHPsDls&s=TJmubZ_4H7iiYRramtlMy84fx1lyLkuOJfA391x78cs&e=
>>>> Archives:
>>>>https://urldefense.proofpoint.com/v2/url?u=http-3A__active4d-2Dnabble.a
>>>>parajitaworld.com_&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8UQmO8P5M
>>>>3-rwSBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wobVcHPsD
>>>>ls&s=VJzLTdVZBbjHRZH1usEIxPXoRTc4FrJF5QANN_xTdRk&e=
>>>
>>>
>>> _______________________________________________
>>> Active4D-dev mailing list
>>> [hidden email]
>>>
>>>https://urldefense.proofpoint.com/v2/url?u=http-3A__list.aparajitaworld.
>>>com_listinfo_active4d-2Ddev&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8
>>>UQmO8P5M3-rwSBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wo
>>>bVcHPsDls&s=TJmubZ_4H7iiYRramtlMy84fx1lyLkuOJfA391x78cs&e=
>>> Archives:
>>>https://urldefense.proofpoint.com/v2/url?u=http-3A__active4d-2Dnabble.ap
>>>arajitaworld.com_&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8UQmO8P5M3-
>>>rwSBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wobVcHPsDls&
>>>s=VJzLTdVZBbjHRZH1usEIxPXoRTc4FrJF5QANN_xTdRk&e=
>>
>> _______________________________________________
>> Active4D-dev mailing list
>> [hidden email]
>>
>>https://urldefense.proofpoint.com/v2/url?u=http-3A__list.aparajitaworld.c
>>om_listinfo_active4d-2Ddev&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8UQ
>>mO8P5M3-rwSBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wobVc
>>HPsDls&s=TJmubZ_4H7iiYRramtlMy84fx1lyLkuOJfA391x78cs&e=
>> Archives:
>>https://urldefense.proofpoint.com/v2/url?u=http-3A__active4d-2Dnabble.apa
>>rajitaworld.com_&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8UQmO8P5M3-rw
>>SBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wobVcHPsDls&s=V
>>JzLTdVZBbjHRZH1usEIxPXoRTc4FrJF5QANN_xTdRk&e=
>
>
>_______________________________________________
>Active4D-dev mailing list
>[hidden email]
>https://urldefense.proofpoint.com/v2/url?u=http-3A__list.aparajitaworld.co
>m_listinfo_active4d-2Ddev&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8UQmO
>8P5M3-rwSBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wobVcHPs
>Dls&s=TJmubZ_4H7iiYRramtlMy84fx1lyLkuOJfA391x78cs&e=
>Archives:
>https://urldefense.proofpoint.com/v2/url?u=http-3A__active4d-2Dnabble.apar
>ajitaworld.com_&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8UQmO8P5M3-rwSB
>mptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wobVcHPsDls&s=VJzL
>TdVZBbjHRZH1usEIxPXoRTc4FrJF5QANN_xTdRk&e=
>_______________________________________________
>Active4D-dev mailing list
>[hidden email]
>http://list.aparajitaworld.com/listinfo/active4d-dev
>Archives: http://active4d-nabble.aparajitaworld.com/

_______________________________________________
Active4D-dev mailing list
[hidden email]
http://list.aparajitaworld.com/listinfo/active4d-dev
Archives: http://active4d-nabble.aparajitaworld.com/
Reply | Threaded
Open this post in threaded view
|

Re: Active4D Slowdown then Stall (weird A4D_HTTPD_RequestHandler_* behavior)

Aparajita Fishman
Brad,

Is it possible to roll back to the old NTK server code? I'd like to know if that solves the problem.

All the best,

- Aparajita

> On Apr 24, 2017, at 1:53 PM, Perkins, Bradley D <[hidden email]> wrote:
>
> Hi John,
>
> We went from v14 to v15. We're also still running 32-bit server. I'm not
> positive but I thought the new network layer was only available in 64-bit.
> I will definitely check though. Thanks!
>
> Brad
>
> On 4/24/17, 2:31 PM, "Active4D-dev on behalf of Bellos, John"
> <[hidden email] on behalf of
> [hidden email]> wrote:
>
>> I forget what version of 4D you said you came from, but if it was v13 or
>> lower, try your setup with the new network layer OFF in 4D server. We've
>> definitely seen some stability issues with the new network layer ON,
>> using v15.4.
>>
>> Note that if you're using Mac 64-Bit, the new network layer is forced on
>> and you can't turn it off. But on Windows 64 bit server, you can turn
>> this setting off.
>>
>> ________________________________________
>> From: Active4D-dev [[hidden email]] on behalf of
>> Aparajita Fishman [[hidden email]]
>> Sent: Monday, April 24, 2017 4:29 PM
>> To: Active4D Developer Discussion List
>> Subject: Re: [Active4d-dev] Active4D Slowdown then Stall (weird
>> A4D_HTTPD_RequestHandler_* behavior)
>>
>> It’s entirely possible there is some flaw in the new server code. The
>> main change I made to A4D_HTTPD_RequestHandler was the correct handling
>> of keep-alive connections. If you find out any more details, please post
>> them here.
>>
>> All the best,
>>
>> - Aparajita
>>
>>> On Apr 24, 2017, at 8:43 AM, Perkins, Bradley D <[hidden email]>
>>> wrote:
>>>
>>> Peter and Aparajita,
>>>
>>> Thank you both for your responses. I have been testing our development
>>> system with a higher MaxThreads this morning. I haven't been able to
>>> exhaust all of the threads. I was able to crash 4D through. I'll send
>>> Aparajita the crash log.
>>>
>>> Peter, your observations are very similar to what we've seen. Under v14
>>> --
>>> and even with the broken keep alive feature -- we could run for a long
>>> time with no problems. Reboots were usually because I had to reboot the
>>> Windows server to install MS Security Patches.
>>>
>>> Our production system sees a fairly light load and would normally never
>>> see the loads that I'm placing on it with ApacheBench unless IT is
>>> running
>>> a cyber scan. I'm letting it run as is with the earlier configuration to
>>> see if the problem is because we've exhausted the thread pool. We
>>> haven't
>>> allocated an additional thread since I restarted on Saturday. Our
>>> ThreadStack size was already set to 256 (docs show 128 as default).
>>> As Peter notes, It could be that the problem lies elsewhere such as in
>>> in
>>> 4D itself.
>>>
>>> We upgraded hardware, OS, 4D, NTK, and Active4D. That is a lot of
>>> change.
>>>
>>> -- Brad
>>>
>>> On 4/24/17, 9:09 AM, "Active4D-dev on behalf of Aparajita Fishman"
>>> <[hidden email] on behalf of
>>> [hidden email]> wrote:
>>>
>>>> Increase MaxThreads in the A4D_NTKConfig list. I am increasing this to
>>>> 100 by default in the next release, and I am also adding code to
>>>> reclaim
>>>> unused threads after a configurable delay.
>>>>
>>>> All the best,
>>>>
>>>> - Aparajita
>>>>
>>>>> On Apr 24, 2017, at 4:51 AM, Peter Gutbrod <[hidden email]>
>>>>> wrote:
>>>>>
>>>>> Hi Brad,
>>>>>
>>>>> I see something similar since I've upgraded 4D-Server to V16 and
>>>>> NTK/A4D
>>>>> to the newest version.
>>>>>
>>>>> Before the server could run für month without any issue. After the
>>>>> upgrade the server slows down within 1-2 days up the point, where each
>>>>> request can take 10 seconds or even more. According to the 4D admin
>>>>> panel there are cpu or network intensive tasks running. Server is
>>>>> almost
>>>>> idle, tons af free memory for 4D. Nothing suspicious in Windows
>>>>> performance monitor. Rebooting 4D and the slow-down is gone -  for the
>>>>> next couple of hours, then it slowly reappears.
>>>>>
>>>>> I realized the slowdown in a JSON API. First I thought it is something
>>>>> within my A4D code, so I've calculated the time my A4D code needs to
>>>>> run. But I found the slowdown happens while receiving the request.
>>>>> Once
>>>>> A4D starts executing the code, everything runs fast, including sending
>>>>> the response.
>>>>>
>>>>> I've asked Rob from the pluggers, whether he had heard about slowdows
>>>>> with the new version of NTK but he wasn't aware of any problems.
>>>>>
>>>>> In a period, where I wasn't able to restart 4D server periodically,
>>>>> slowdown got so extreme, that 4D clients were unable to login due to
>>>>> timeouts. In that state the only way to restart the 4D server was
>>>>> with a
>>>>> force quit. That convinced me, that the culprit is within 4D V16 and
>>>>> not
>>>>> NTK or A4D and I should wait for 4D 16.1 and until then restart 4D
>>>>> Server periodically. That is why I never reported the problem here.
>>>>>
>>>>> As you have a similar problem but with 4D15.4 makes me wondering,
>>>>> whether I'm wrong and the lastest versions of NTK and A4D might at
>>>>> least
>>>>> contribute to the problem.
>>>>>
>>>>> Hope you find the answer and share it with me.
>>>>>
>>>>> Greetings
>>>>>
>>>>> Peter Gutbrod
>>>>> _______________________________________________
>>>>> Active4D-dev mailing list
>>>>> [hidden email]
>>>>>
>>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__list.aparajitaworld
>>>>> .com_listinfo_active4d-2Ddev&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVR
>>>>> q8UQmO8P5M3-rwSBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC
>>>>> 6wobVcHPsDls&s=TJmubZ_4H7iiYRramtlMy84fx1lyLkuOJfA391x78cs&e=
>>>>> Archives:
>>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__active4d-2Dnabble.a
>>>>> parajitaworld.com_&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8UQmO8P5M
>>>>> 3-rwSBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wobVcHPsD
>>>>> ls&s=VJzLTdVZBbjHRZH1usEIxPXoRTc4FrJF5QANN_xTdRk&e=
>>>>
>>>>
>>>> _______________________________________________
>>>> Active4D-dev mailing list
>>>> [hidden email]
>>>>
>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__list.aparajitaworld.
>>>> com_listinfo_active4d-2Ddev&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8
>>>> UQmO8P5M3-rwSBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wo
>>>> bVcHPsDls&s=TJmubZ_4H7iiYRramtlMy84fx1lyLkuOJfA391x78cs&e=
>>>> Archives:
>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__active4d-2Dnabble.ap
>>>> arajitaworld.com_&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8UQmO8P5M3-
>>>> rwSBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wobVcHPsDls&
>>>> s=VJzLTdVZBbjHRZH1usEIxPXoRTc4FrJF5QANN_xTdRk&e=
>>>
>>> _______________________________________________
>>> Active4D-dev mailing list
>>> [hidden email]
>>>
>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__list.aparajitaworld.c
>>> om_listinfo_active4d-2Ddev&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8UQ
>>> mO8P5M3-rwSBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wobVc
>>> HPsDls&s=TJmubZ_4H7iiYRramtlMy84fx1lyLkuOJfA391x78cs&e=
>>> Archives:
>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__active4d-2Dnabble.apa
>>> rajitaworld.com_&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8UQmO8P5M3-rw
>>> SBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wobVcHPsDls&s=V
>>> JzLTdVZBbjHRZH1usEIxPXoRTc4FrJF5QANN_xTdRk&e=
>>
>>
>> _______________________________________________
>> Active4D-dev mailing list
>> [hidden email]
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__list.aparajitaworld.co
>> m_listinfo_active4d-2Ddev&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8UQmO
>> 8P5M3-rwSBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wobVcHPs
>> Dls&s=TJmubZ_4H7iiYRramtlMy84fx1lyLkuOJfA391x78cs&e=
>> Archives:
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__active4d-2Dnabble.apar
>> ajitaworld.com_&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8UQmO8P5M3-rwSB
>> mptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wobVcHPsDls&s=VJzL
>> TdVZBbjHRZH1usEIxPXoRTc4FrJF5QANN_xTdRk&e=
>> _______________________________________________
>> Active4D-dev mailing list
>> [hidden email]
>> http://list.aparajitaworld.com/listinfo/active4d-dev
>> Archives: http://active4d-nabble.aparajitaworld.com/
>
> _______________________________________________
> Active4D-dev mailing list
> [hidden email]
> http://list.aparajitaworld.com/listinfo/active4d-dev
> Archives: http://active4d-nabble.aparajitaworld.com/


_______________________________________________
Active4D-dev mailing list
[hidden email]
http://list.aparajitaworld.com/listinfo/active4d-dev
Archives: http://active4d-nabble.aparajitaworld.com/
Reply | Threaded
Open this post in threaded view
|

Re: Active4D Slowdown then Stall (weird A4D_HTTPD_RequestHandler_* behavior)

Perkins, Bradley D
Not really. I'd have to:
(a) Go back to the old server machine and 4D v14
(b) Work with institutional IT to get it back on the network (Hostmaster
updates, Switch programming). They tightly control all public facing IP
addresses.
(c) Identify all changed data since we went live and export/import it back
into the v14 Database.
As noted that ran fine.

I never considered running v14 on the Sierra machines. I was told by 4D it
wouldn't be stable.

If you mean staying with v15.4 and putting in the old shell code and going
back to older NTK and Active4D versions, we could try that, but I have
other other project commitments that I must focus on for the next few
weeks and can't work on that now. We're just going to have to monitor our
systems and live with the problems if they persist.

One idea I did have was to compile the NTK demo database under v15.4
running same NTK and A4D versions and place that on our development
system, reasonably exercise it with AB, and see if the Active4D/NTK server
becomes unresponsive.

Also, I checked and we are still using the old network layer.


Thanks,

-- Brad

On 4/25/17, 7:30 AM, "Active4D-dev on behalf of Aparajita Fishman"
<[hidden email] on behalf of
[hidden email]> wrote:

>Brad,
>
>Is it possible to roll back to the old NTK server code? I'd like to know
>if that solves the problem.
>
>All the best,
>
>- Aparajita
>
>> On Apr 24, 2017, at 1:53 PM, Perkins, Bradley D <[hidden email]>
>>wrote:
>>
>> Hi John,
>>
>> We went from v14 to v15. We're also still running 32-bit server. I'm not
>> positive but I thought the new network layer was only available in
>>64-bit.
>> I will definitely check though. Thanks!
>>
>> Brad
>>
>> On 4/24/17, 2:31 PM, "Active4D-dev on behalf of Bellos, John"
>> <[hidden email] on behalf of
>> [hidden email]> wrote:
>>
>>> I forget what version of 4D you said you came from, but if it was v13
>>>or
>>> lower, try your setup with the new network layer OFF in 4D server.
>>>We've
>>> definitely seen some stability issues with the new network layer ON,
>>> using v15.4.
>>>
>>> Note that if you're using Mac 64-Bit, the new network layer is forced
>>>on
>>> and you can't turn it off. But on Windows 64 bit server, you can turn
>>> this setting off.
>>>
>>> ________________________________________
>>> From: Active4D-dev [[hidden email]] on behalf
>>>of
>>> Aparajita Fishman [[hidden email]]
>>> Sent: Monday, April 24, 2017 4:29 PM
>>> To: Active4D Developer Discussion List
>>> Subject: Re: [Active4d-dev] Active4D Slowdown then Stall (weird
>>> A4D_HTTPD_RequestHandler_* behavior)
>>>
>>> It’s entirely possible there is some flaw in the new server code. The
>>> main change I made to A4D_HTTPD_RequestHandler was the correct handling
>>> of keep-alive connections. If you find out any more details, please
>>>post
>>> them here.
>>>
>>> All the best,
>>>
>>> - Aparajita
>>>
>>>> On Apr 24, 2017, at 8:43 AM, Perkins, Bradley D <[hidden email]>
>>>> wrote:
>>>>
>>>> Peter and Aparajita,
>>>>
>>>> Thank you both for your responses. I have been testing our development
>>>> system with a higher MaxThreads this morning. I haven't been able to
>>>> exhaust all of the threads. I was able to crash 4D through. I'll send
>>>> Aparajita the crash log.
>>>>
>>>> Peter, your observations are very similar to what we've seen. Under
>>>>v14
>>>> --
>>>> and even with the broken keep alive feature -- we could run for a long
>>>> time with no problems. Reboots were usually because I had to reboot
>>>>the
>>>> Windows server to install MS Security Patches.
>>>>
>>>> Our production system sees a fairly light load and would normally
>>>>never
>>>> see the loads that I'm placing on it with ApacheBench unless IT is
>>>> running
>>>> a cyber scan. I'm letting it run as is with the earlier configuration
>>>>to
>>>> see if the problem is because we've exhausted the thread pool. We
>>>> haven't
>>>> allocated an additional thread since I restarted on Saturday. Our
>>>> ThreadStack size was already set to 256 (docs show 128 as default).
>>>> As Peter notes, It could be that the problem lies elsewhere such as in
>>>> in
>>>> 4D itself.
>>>>
>>>> We upgraded hardware, OS, 4D, NTK, and Active4D. That is a lot of
>>>> change.
>>>>
>>>> -- Brad
>>>>
>>>> On 4/24/17, 9:09 AM, "Active4D-dev on behalf of Aparajita Fishman"
>>>> <[hidden email] on behalf of
>>>> [hidden email]> wrote:
>>>>
>>>>> Increase MaxThreads in the A4D_NTKConfig list. I am increasing this
>>>>>to
>>>>> 100 by default in the next release, and I am also adding code to
>>>>> reclaim
>>>>> unused threads after a configurable delay.
>>>>>
>>>>> All the best,
>>>>>
>>>>> - Aparajita
>>>>>
>>>>>> On Apr 24, 2017, at 4:51 AM, Peter Gutbrod <[hidden email]>
>>>>>> wrote:
>>>>>>
>>>>>> Hi Brad,
>>>>>>
>>>>>> I see something similar since I've upgraded 4D-Server to V16 and
>>>>>> NTK/A4D
>>>>>> to the newest version.
>>>>>>
>>>>>> Before the server could run für month without any issue. After the
>>>>>> upgrade the server slows down within 1-2 days up the point, where
>>>>>>each
>>>>>> request can take 10 seconds or even more. According to the 4D admin
>>>>>> panel there are cpu or network intensive tasks running. Server is
>>>>>> almost
>>>>>> idle, tons af free memory for 4D. Nothing suspicious in Windows
>>>>>> performance monitor. Rebooting 4D and the slow-down is gone -  for
>>>>>>the
>>>>>> next couple of hours, then it slowly reappears.
>>>>>>
>>>>>> I realized the slowdown in a JSON API. First I thought it is
>>>>>>something
>>>>>> within my A4D code, so I've calculated the time my A4D code needs to
>>>>>> run. But I found the slowdown happens while receiving the request.
>>>>>> Once
>>>>>> A4D starts executing the code, everything runs fast, including
>>>>>>sending
>>>>>> the response.
>>>>>>
>>>>>> I've asked Rob from the pluggers, whether he had heard about
>>>>>>slowdows
>>>>>> with the new version of NTK but he wasn't aware of any problems.
>>>>>>
>>>>>> In a period, where I wasn't able to restart 4D server periodically,
>>>>>> slowdown got so extreme, that 4D clients were unable to login due to
>>>>>> timeouts. In that state the only way to restart the 4D server was
>>>>>> with a
>>>>>> force quit. That convinced me, that the culprit is within 4D V16 and
>>>>>> not
>>>>>> NTK or A4D and I should wait for 4D 16.1 and until then restart 4D
>>>>>> Server periodically. That is why I never reported the problem here.
>>>>>>
>>>>>> As you have a similar problem but with 4D15.4 makes me wondering,
>>>>>> whether I'm wrong and the lastest versions of NTK and A4D might at
>>>>>> least
>>>>>> contribute to the problem.
>>>>>>
>>>>>> Hope you find the answer and share it with me.
>>>>>>
>>>>>> Greetings
>>>>>>
>>>>>> Peter Gutbrod
>>>>>> _______________________________________________
>>>>>> Active4D-dev mailing list
>>>>>> [hidden email]
>>>>>>
>>>>>>
>>>>>>https://urldefense.proofpoint.com/v2/url?u=http-3A__list.aparajitawor
>>>>>>ld
>>>>>>
>>>>>>.com_listinfo_active4d-2Ddev&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtj
>>>>>>VR
>>>>>>
>>>>>>q8UQmO8P5M3-rwSBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PK
>>>>>>YC
>>>>>> 6wobVcHPsDls&s=TJmubZ_4H7iiYRramtlMy84fx1lyLkuOJfA391x78cs&e=
>>>>>> Archives:
>>>>>>
>>>>>>https://urldefense.proofpoint.com/v2/url?u=http-3A__active4d-2Dnabble
>>>>>>.a
>>>>>>
>>>>>>parajitaworld.com_&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8UQmO8P
>>>>>>5M
>>>>>>
>>>>>>3-rwSBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wobVcHP
>>>>>>sD
>>>>>> ls&s=VJzLTdVZBbjHRZH1usEIxPXoRTc4FrJF5QANN_xTdRk&e=
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Active4D-dev mailing list
>>>>> [hidden email]
>>>>>
>>>>>
>>>>>https://urldefense.proofpoint.com/v2/url?u=http-3A__list.aparajitaworl
>>>>>d.
>>>>>
>>>>>com_listinfo_active4d-2Ddev&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVR
>>>>>q8
>>>>>
>>>>>UQmO8P5M3-rwSBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6
>>>>>wo
>>>>> bVcHPsDls&s=TJmubZ_4H7iiYRramtlMy84fx1lyLkuOJfA391x78cs&e=
>>>>> Archives:
>>>>>
>>>>>https://urldefense.proofpoint.com/v2/url?u=http-3A__active4d-2Dnabble.
>>>>>ap
>>>>>
>>>>>arajitaworld.com_&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8UQmO8P5M
>>>>>3-
>>>>>
>>>>>rwSBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wobVcHPsDl
>>>>>s&
>>>>> s=VJzLTdVZBbjHRZH1usEIxPXoRTc4FrJF5QANN_xTdRk&e=
>>>>
>>>> _______________________________________________
>>>> Active4D-dev mailing list
>>>> [hidden email]
>>>>
>>>>
>>>>https://urldefense.proofpoint.com/v2/url?u=http-3A__list.aparajitaworld
>>>>.c
>>>>
>>>>om_listinfo_active4d-2Ddev&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8
>>>>UQ
>>>>
>>>>mO8P5M3-rwSBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wob
>>>>Vc
>>>> HPsDls&s=TJmubZ_4H7iiYRramtlMy84fx1lyLkuOJfA391x78cs&e=
>>>> Archives:
>>>>
>>>>https://urldefense.proofpoint.com/v2/url?u=http-3A__active4d-2Dnabble.a
>>>>pa
>>>>
>>>>rajitaworld.com_&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8UQmO8P5M3-
>>>>rw
>>>>
>>>>SBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wobVcHPsDls&s
>>>>=V
>>>> JzLTdVZBbjHRZH1usEIxPXoRTc4FrJF5QANN_xTdRk&e=
>>>
>>>
>>> _______________________________________________
>>> Active4D-dev mailing list
>>> [hidden email]
>>>
>>>https://urldefense.proofpoint.com/v2/url?u=http-3A__list.aparajitaworld.
>>>co
>>>
>>>m_listinfo_active4d-2Ddev&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8UQ
>>>mO
>>>
>>>8P5M3-rwSBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wobVcH
>>>Ps
>>> Dls&s=TJmubZ_4H7iiYRramtlMy84fx1lyLkuOJfA391x78cs&e=
>>> Archives:
>>>
>>>https://urldefense.proofpoint.com/v2/url?u=http-3A__active4d-2Dnabble.ap
>>>ar
>>>
>>>ajitaworld.com_&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8UQmO8P5M3-rw
>>>SB
>>>
>>>mptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wobVcHPsDls&s=VJ
>>>zL
>>> TdVZBbjHRZH1usEIxPXoRTc4FrJF5QANN_xTdRk&e=
>>> _______________________________________________
>>> Active4D-dev mailing list
>>> [hidden email]
>>> http://list.aparajitaworld.com/listinfo/active4d-dev
>>> Archives: http://active4d-nabble.aparajitaworld.com/
>>
>> _______________________________________________
>> Active4D-dev mailing list
>> [hidden email]
>> http://list.aparajitaworld.com/listinfo/active4d-dev
>> Archives: http://active4d-nabble.aparajitaworld.com/
>
>
>_______________________________________________
>Active4D-dev mailing list
>[hidden email]
>http://list.aparajitaworld.com/listinfo/active4d-dev
>Archives: http://active4d-nabble.aparajitaworld.com/

_______________________________________________
Active4D-dev mailing list
[hidden email]
http://list.aparajitaworld.com/listinfo/active4d-dev
Archives: http://active4d-nabble.aparajitaworld.com/
Reply | Threaded
Open this post in threaded view
|

Re: Active4D Slowdown then Stall (weird A4D_HTTPD_RequestHandler_* behavior)

Aparajita Fishman
I meant only to use the old NTK shell code. Nothing else would have to change. If your problems persist and I can’t find the solution, I’ll do it for free.

All the best,

  - Aparajita

> On Apr 25, 2017, at 7:21 AM, Perkins, Bradley D <[hidden email]> wrote:
>
> Not really. I'd have to:
> (a) Go back to the old server machine and 4D v14
> (b) Work with institutional IT to get it back on the network (Hostmaster
> updates, Switch programming). They tightly control all public facing IP
> addresses.
> (c) Identify all changed data since we went live and export/import it back
> into the v14 Database.
> As noted that ran fine.
>
> I never considered running v14 on the Sierra machines. I was told by 4D it
> wouldn't be stable.
>
> If you mean staying with v15.4 and putting in the old shell code and going
> back to older NTK and Active4D versions, we could try that, but I have
> other other project commitments that I must focus on for the next few
> weeks and can't work on that now. We're just going to have to monitor our
> systems and live with the problems if they persist.
>
> One idea I did have was to compile the NTK demo database under v15.4
> running same NTK and A4D versions and place that on our development
> system, reasonably exercise it with AB, and see if the Active4D/NTK server
> becomes unresponsive.
>
> Also, I checked and we are still using the old network layer.
>
>
> Thanks,
>
> -- Brad
>
> On 4/25/17, 7:30 AM, "Active4D-dev on behalf of Aparajita Fishman"
> <[hidden email] on behalf of
> [hidden email]> wrote:
>
>> Brad,
>>
>> Is it possible to roll back to the old NTK server code? I'd like to know
>> if that solves the problem.
>>
>> All the best,
>>
>> - Aparajita
>>
>>> On Apr 24, 2017, at 1:53 PM, Perkins, Bradley D <[hidden email]>
>>> wrote:
>>>
>>> Hi John,
>>>
>>> We went from v14 to v15. We're also still running 32-bit server. I'm not
>>> positive but I thought the new network layer was only available in
>>> 64-bit.
>>> I will definitely check though. Thanks!
>>>
>>> Brad
>>>
>>> On 4/24/17, 2:31 PM, "Active4D-dev on behalf of Bellos, John"
>>> <[hidden email] on behalf of
>>> [hidden email]> wrote:
>>>
>>>> I forget what version of 4D you said you came from, but if it was v13
>>>> or
>>>> lower, try your setup with the new network layer OFF in 4D server.
>>>> We've
>>>> definitely seen some stability issues with the new network layer ON,
>>>> using v15.4.
>>>>
>>>> Note that if you're using Mac 64-Bit, the new network layer is forced
>>>> on
>>>> and you can't turn it off. But on Windows 64 bit server, you can turn
>>>> this setting off.
>>>>
>>>> ________________________________________
>>>> From: Active4D-dev [[hidden email]] on behalf
>>>> of
>>>> Aparajita Fishman [[hidden email]]
>>>> Sent: Monday, April 24, 2017 4:29 PM
>>>> To: Active4D Developer Discussion List
>>>> Subject: Re: [Active4d-dev] Active4D Slowdown then Stall (weird
>>>> A4D_HTTPD_RequestHandler_* behavior)
>>>>
>>>> It’s entirely possible there is some flaw in the new server code. The
>>>> main change I made to A4D_HTTPD_RequestHandler was the correct handling
>>>> of keep-alive connections. If you find out any more details, please
>>>> post
>>>> them here.
>>>>
>>>> All the best,
>>>>
>>>> - Aparajita
>>>>
>>>>> On Apr 24, 2017, at 8:43 AM, Perkins, Bradley D <[hidden email]>
>>>>> wrote:
>>>>>
>>>>> Peter and Aparajita,
>>>>>
>>>>> Thank you both for your responses. I have been testing our development
>>>>> system with a higher MaxThreads this morning. I haven't been able to
>>>>> exhaust all of the threads. I was able to crash 4D through. I'll send
>>>>> Aparajita the crash log.
>>>>>
>>>>> Peter, your observations are very similar to what we've seen. Under
>>>>> v14
>>>>> --
>>>>> and even with the broken keep alive feature -- we could run for a long
>>>>> time with no problems. Reboots were usually because I had to reboot
>>>>> the
>>>>> Windows server to install MS Security Patches.
>>>>>
>>>>> Our production system sees a fairly light load and would normally
>>>>> never
>>>>> see the loads that I'm placing on it with ApacheBench unless IT is
>>>>> running
>>>>> a cyber scan. I'm letting it run as is with the earlier configuration
>>>>> to
>>>>> see if the problem is because we've exhausted the thread pool. We
>>>>> haven't
>>>>> allocated an additional thread since I restarted on Saturday. Our
>>>>> ThreadStack size was already set to 256 (docs show 128 as default).
>>>>> As Peter notes, It could be that the problem lies elsewhere such as in
>>>>> in
>>>>> 4D itself.
>>>>>
>>>>> We upgraded hardware, OS, 4D, NTK, and Active4D. That is a lot of
>>>>> change.
>>>>>
>>>>> -- Brad
>>>>>
>>>>> On 4/24/17, 9:09 AM, "Active4D-dev on behalf of Aparajita Fishman"
>>>>> <[hidden email] on behalf of
>>>>> [hidden email]> wrote:
>>>>>
>>>>>> Increase MaxThreads in the A4D_NTKConfig list. I am increasing this
>>>>>> to
>>>>>> 100 by default in the next release, and I am also adding code to
>>>>>> reclaim
>>>>>> unused threads after a configurable delay.
>>>>>>
>>>>>> All the best,
>>>>>>
>>>>>> - Aparajita
>>>>>>
>>>>>>> On Apr 24, 2017, at 4:51 AM, Peter Gutbrod <[hidden email]>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi Brad,
>>>>>>>
>>>>>>> I see something similar since I've upgraded 4D-Server to V16 and
>>>>>>> NTK/A4D
>>>>>>> to the newest version.
>>>>>>>
>>>>>>> Before the server could run für month without any issue. After the
>>>>>>> upgrade the server slows down within 1-2 days up the point, where
>>>>>>> each
>>>>>>> request can take 10 seconds or even more. According to the 4D admin
>>>>>>> panel there are cpu or network intensive tasks running. Server is
>>>>>>> almost
>>>>>>> idle, tons af free memory for 4D. Nothing suspicious in Windows
>>>>>>> performance monitor. Rebooting 4D and the slow-down is gone -  for
>>>>>>> the
>>>>>>> next couple of hours, then it slowly reappears.
>>>>>>>
>>>>>>> I realized the slowdown in a JSON API. First I thought it is
>>>>>>> something
>>>>>>> within my A4D code, so I've calculated the time my A4D code needs to
>>>>>>> run. But I found the slowdown happens while receiving the request.
>>>>>>> Once
>>>>>>> A4D starts executing the code, everything runs fast, including
>>>>>>> sending
>>>>>>> the response.
>>>>>>>
>>>>>>> I've asked Rob from the pluggers, whether he had heard about
>>>>>>> slowdows
>>>>>>> with the new version of NTK but he wasn't aware of any problems.
>>>>>>>
>>>>>>> In a period, where I wasn't able to restart 4D server periodically,
>>>>>>> slowdown got so extreme, that 4D clients were unable to login due to
>>>>>>> timeouts. In that state the only way to restart the 4D server was
>>>>>>> with a
>>>>>>> force quit. That convinced me, that the culprit is within 4D V16 and
>>>>>>> not
>>>>>>> NTK or A4D and I should wait for 4D 16.1 and until then restart 4D
>>>>>>> Server periodically. That is why I never reported the problem here.
>>>>>>>
>>>>>>> As you have a similar problem but with 4D15.4 makes me wondering,
>>>>>>> whether I'm wrong and the lastest versions of NTK and A4D might at
>>>>>>> least
>>>>>>> contribute to the problem.
>>>>>>>
>>>>>>> Hope you find the answer and share it with me.
>>>>>>>
>>>>>>> Greetings
>>>>>>>
>>>>>>> Peter Gutbrod
>>>>>>> _______________________________________________
>>>>>>> Active4D-dev mailing list
>>>>>>> [hidden email]
>>>>>>>
>>>>>>>
>>>>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__list.aparajitawor
>>>>>>> ld
>>>>>>>
>>>>>>> .com_listinfo_active4d-2Ddev&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtj
>>>>>>> VR
>>>>>>>
>>>>>>> q8UQmO8P5M3-rwSBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PK
>>>>>>> YC
>>>>>>> 6wobVcHPsDls&s=TJmubZ_4H7iiYRramtlMy84fx1lyLkuOJfA391x78cs&e=
>>>>>>> Archives:
>>>>>>>
>>>>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__active4d-2Dnabble
>>>>>>> .a
>>>>>>>
>>>>>>> parajitaworld.com_&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8UQmO8P
>>>>>>> 5M
>>>>>>>
>>>>>>> 3-rwSBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wobVcHP
>>>>>>> sD
>>>>>>> ls&s=VJzLTdVZBbjHRZH1usEIxPXoRTc4FrJF5QANN_xTdRk&e=
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Active4D-dev mailing list
>>>>>> [hidden email]
>>>>>>
>>>>>>
>>>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__list.aparajitaworl
>>>>>> d.
>>>>>>
>>>>>> com_listinfo_active4d-2Ddev&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVR
>>>>>> q8
>>>>>>
>>>>>> UQmO8P5M3-rwSBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6
>>>>>> wo
>>>>>> bVcHPsDls&s=TJmubZ_4H7iiYRramtlMy84fx1lyLkuOJfA391x78cs&e=
>>>>>> Archives:
>>>>>>
>>>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__active4d-2Dnabble.
>>>>>> ap
>>>>>>
>>>>>> arajitaworld.com_&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8UQmO8P5M
>>>>>> 3-
>>>>>>
>>>>>> rwSBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wobVcHPsDl
>>>>>> s&
>>>>>> s=VJzLTdVZBbjHRZH1usEIxPXoRTc4FrJF5QANN_xTdRk&e=
>>>>>
>>>>> _______________________________________________
>>>>> Active4D-dev mailing list
>>>>> [hidden email]
>>>>>
>>>>>
>>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__list.aparajitaworld
>>>>> .c
>>>>>
>>>>> om_listinfo_active4d-2Ddev&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8
>>>>> UQ
>>>>>
>>>>> mO8P5M3-rwSBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wob
>>>>> Vc
>>>>> HPsDls&s=TJmubZ_4H7iiYRramtlMy84fx1lyLkuOJfA391x78cs&e=
>>>>> Archives:
>>>>>
>>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__active4d-2Dnabble.a
>>>>> pa
>>>>>
>>>>> rajitaworld.com_&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8UQmO8P5M3-
>>>>> rw
>>>>>
>>>>> SBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wobVcHPsDls&s
>>>>> =V
>>>>> JzLTdVZBbjHRZH1usEIxPXoRTc4FrJF5QANN_xTdRk&e=
>>>>
>>>>
>>>> _______________________________________________
>>>> Active4D-dev mailing list
>>>> [hidden email]
>>>>
>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__list.aparajitaworld.
>>>> co
>>>>
>>>> m_listinfo_active4d-2Ddev&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8UQ
>>>> mO
>>>>
>>>> 8P5M3-rwSBmptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wobVcH
>>>> Ps
>>>> Dls&s=TJmubZ_4H7iiYRramtlMy84fx1lyLkuOJfA391x78cs&e=
>>>> Archives:
>>>>
>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__active4d-2Dnabble.ap
>>>> ar
>>>>
>>>> ajitaworld.com_&d=DwIGaQ&c=2do6VJGs3LvEOe4OFFM1bA&r=wwtjVRq8UQmO8P5M3-rw
>>>> SB
>>>>
>>>> mptOUUgRJNGBQlLDclI30&m=WKACtLPtkhCbwM0QslQxK69p4G5PKYC6wobVcHPsDls&s=VJ
>>>> zL
>>>> TdVZBbjHRZH1usEIxPXoRTc4FrJF5QANN_xTdRk&e=
>>>> _______________________________________________
>>>> Active4D-dev mailing list
>>>> [hidden email]
>>>> http://list.aparajitaworld.com/listinfo/active4d-dev
>>>> Archives: http://active4d-nabble.aparajitaworld.com/
>>>
>>> _______________________________________________
>>> Active4D-dev mailing list
>>> [hidden email]
>>> http://list.aparajitaworld.com/listinfo/active4d-dev
>>> Archives: http://active4d-nabble.aparajitaworld.com/
>>
>>
>> _______________________________________________
>> Active4D-dev mailing list
>> [hidden email]
>> http://list.aparajitaworld.com/listinfo/active4d-dev
>> Archives: http://active4d-nabble.aparajitaworld.com/
>
> _______________________________________________
> Active4D-dev mailing list
> [hidden email]
> http://list.aparajitaworld.com/listinfo/active4d-dev
> Archives: http://active4d-nabble.aparajitaworld.com/


_______________________________________________
Active4D-dev mailing list
[hidden email]
http://list.aparajitaworld.com/listinfo/active4d-dev
Archives: http://active4d-nabble.aparajitaworld.com/
Reply | Threaded
Open this post in threaded view
|

Re: Active4D Slowdown then Stall (weird A4D_HTTPD_RequestHandler_* behavior)

Peter Gutbrod
In reply to this post by Aparajita Fishman
I had set from the beginning:

NumThreads=13
MaxThreads=97
ThreadStack=256
CompressResponse=1
KeepAliveTime=5

Don't think MaxThreads=97 would make a difference.

Any other values to play with?

Rebooted early in the morning and 12 hours later there are still no
slowdowns, fingers crossed.

Peter

> Aparajita Fishman <mailto:[hidden email]>
> 24. April 2017 um 17:09
> Increase MaxThreads in the A4D_NTKConfig list. I am increasing this to
> 100 by default in the next release, and I am also adding code to
> reclaim unused threads after a configurable delay.
>
> All the best,
>
> - Aparajita
>
>
>
> _______________________________________________
> Active4D-dev mailing list
> [hidden email]
> http://list.aparajitaworld.com/listinfo/active4d-dev
> Archives: http://active4d-nabble.aparajitaworld.com/

_______________________________________________
Active4D-dev mailing list
[hidden email]
http://list.aparajitaworld.com/listinfo/active4d-dev
Archives: http://active4d-nabble.aparajitaworld.com/
12