Question

23

Can I stop wget creating duplicates?

rated 0 times [ 23] [ 0] / answers: 1 / hits: 24436 / 2 Years ago, sun, september 18, 2022, 6:33:43

If I run wget twice, it doesn't recognise that it has already downloaded that file, and creates a new one. Is there any way to prevent it downloading the file again?

$ wget https://cdn.sstatic.net/askubuntu/img/logo.png

...

Saving to: ‘logo.png’

...



$ wget https://cdn.sstatic.net/askubuntu/img/logo.png

...

Saving to: ‘logo.png.1’

...

(Happy to use curl or a similar scriptable alternative if wget can't do this.)

Answers

Only authorized users can answer the question. Please sign in first, or register a free account.

emuralm

Add To Favorites

Follow

Total Points: 290

Total Questions: 111

Total Answers: 117

Location: China

Member since Thu, Sep 1, 2022

2 Years ago

answered 2 Years ago ranquctive · Accepted Answer

I suggest you use the -N option.

-N

--timestamping

    Turn on time-stamping.

It enables time-stamping, which re-downloads the file only if its newer on the server than the downloaded version.

$ wget -N https://cdn.sstatic.net/askubuntu/img/logo.png

...

Saving to: ‘logo.png’

...



$ wget -N https://cdn.sstatic.net/askubuntu/img/logo.png

...

Server file no newer than local file ‘logo.png’ -- not retrieving.

Caveat (from αғsнιη's comment)

If the server is not configured properly, it may always report that the file is new and -N will always re-download the file. In this case, -nc is probably a better option.