I have a site that is like a shopping where the products displayed come from xml files. These xml files are provided by another website, and I need to download them to my server once a week (at least). There are almost 11500 files.

So, I made a script in Python that download all these 11500 xml files over http and save them to my server. The script uses threads, and until last month it was running ok. But now, everytime I execute it, the script get killed. I tried reducing the number of threads, downloading only a few files each time, etc. but nothing seems to work.

Is there anything I can do to avoing this problem? What are the limitations of the server (is it the running time or the amount of processing that is too much)?

Thanks for any tips.

Have you tried to "nice +20" the command to make it run as a lower priority? I've had luck with this on processes that have been killed.

-Scott

I tried "nice -n 19 python my_script.py", but didn't work. Thanks, I emailed the support.

its nice -n+19 not nice -n +19

it doesnt have a "space" between

--
BUGabundo :o)
(``-_-´´)
http://BrinKadeiraS.BUGabundo.net
at http://www.dreamhost.com/r.cgi?249195
50$ discount with promo code "BUG50" on ALL plans

In reply to:

---

its nice -n 19 not nice -n 19
it doesnt have a "space" between

---

You're thinking of "renice". The "nice" command with the "-n 19" is (as posted by the OP) the proper syntax to lower the priority (if you want to raise the priority it'd be "-n -19").
And judging from the first post, this is likely a matter for support; the only alternative I can (currently) think of is to cycle through some reduced selection of the product files being downloaded on a daily basis to spread the updates over a longer period of time.

for some reason your quote of my reply didnt come right.
looking at my "man nice" :
[code]SYNOPSIS
nice [OPTION] [COMMAND [ARG]...]

DESCRIPTION
Run COMMAND with an adjusted niceness, which affects process schedul‐
ing. With no COMMAND, print the current niceness. Nicenesses range
from -20 (most favorable scheduling) to 19 (least favorable).

-n, --adjustment=N
add integer N to the niceness (default 10)
[/code]

--
BUGabundo :o)
(``-_-´´)
http://BrinKadeiraS.BUGabundo.net
at http://www.dreamhost.com/r.cgi?249195
50$ discount with promo code "BUG50" on ALL plans

In reply to:

---

for some reason your quote of my reply didnt come right.

---

Yah, I keep forgetting that this messaging software doesn't like raw plus signs. My bad.
Still, the info is correct; "nice" will lower a process priority if provided with an unsigned integer; save the plus signs for "renice".