Support more of wget's options

[tdulcet]

Since wcurl is being shipped as a replacement for wget by some distributions, consider supporting more of wget's options that have an equivalent option in cURL or could be easily emulated by a shell script. From comparing the wget and cURL man pages, I provided a draft conversion table below, which includes over 80 options. The data is from cURL 8.16.0 and Wget 1.25.0. This obviously would not cover all of the use cases of wget, but hopefully a large percentage. It might be useful to also include a table like this in the cURL documentation to help users with migrating.

Wget options that have an equivalent in cURL

Wget option	cURL option
-V, --version	-V, --version
-h, --help	-h, --help
-q, --quiet	-s, --silent
-v, --verbose	-v, --verbose
-nv, --no-verbose	-sS, --silent --show-error (or --no-progress-meter)
-i file, --input-file=file	--url @filename
--input-metalink=file	--metalink file (no longer supported)
--xattr	--xattr
--bind-dns-address=ADDRESS	--dns-ipv4-addr <address> or --dns-ipv6-addr <address>
--dns-servers=ADDRESSES	--dns-servers <addresses>
-t number, --tries=number	--retry <num> (default 20)
-O file, --output-document=file	-o <file>, --output <file> (per URL, default 'index.html')
-nc, --no-clobber	--skip-existing
--no-netrc	(do not set the -n, --netrc option)
-c, --continue	-C -, --continue-at -
--start-pos=OFFSET	-C, --continue-at <offset> (needs to convert byte suffixes)
--show-progress	-#, --progress-bar
-N, --timestamping	-z <file>, --time-cond <file> (file should exist)
--no-use-server-timestamps	(do not set the -R, --remote-time option)
-S, --server-response	-D -, --dump-header -
--spider	-I, --head
-T seconds, --timeout=seconds	--connect-timeout <seconds> (no DNS or read timeouts)
--connect-timeout=seconds	--connect-timeout <seconds>
--limit-rate=amount	--limit-rate <speed>
--waitretry=seconds	--retry-max-time <seconds> (default 10)
--no-proxy	--noproxy '*'
-4, --inet4-only	-4, --ipv4
-6, --inet6-only	-6, --ipv6
--retry-connrefused	--retry-connrefused
--user=user --password=password	-u <user:password>, --user <user:password>
--user=user --ask-password	-u <user>, --user <user>
-P prefix, --directory-prefix=prefix	--output-dir <dir>
--default-page=name	(change the default -o <file>, --output <file> value)
--http-user=user --http-password=password	(add user and password to any HTTP URLs)
--no-http-keep-alive	--no-keepalive
--no-cookies	(do not set the -b '', --cookie '' option)
--load-cookies file	-b <filename>, --cookie <filename>
--save-cookies file	-c <filename>, --cookie-jar <filename> and -j, --junk-session-cookies
--keep-session-cookies	(do not set the -j, --junk-session-cookies option)
--ignore-length	--ignore-content-length
--header=header-line	-H <header>, --header <header>
--compression=type	--compressed (only if type is not 'none')
--max-redirect=number	--max-redirs <num> (default 20)
--proxy-user=user --proxy-password=password	-U <user:password>, --proxy-user <user:password>
--referer=url	-e <URL>, --referer <URL>
--save-headers	-i, --include, --show-headers
-U agent-string, --user-agent=agent-string	-A <name>, --user-agent <name>
--post-data=string	--data-binary <data>
--post-file=file	--data-binary @<file>
--method=HTTP-Method	-X <method>, --request <method>
--body-data=Data-String	--data-binary <data>
--body-file=Data-File	--data-binary @<file>
--content-disposition	-J, --remote-header-name
--content-on-error	--fail-with-body (instead of the -f, --fail option)
--auth-no-challenge	--basic
--secure-protocol=SSLv2	-2, --sslv2
--secure-protocol=SSLv3	-3, --sslv3
--secure-protocol=TLSv1	--tlsv1.0
--secure-protocol=TLSv1_1	--tlsv1.1
--secure-protocol=TLSv1_2	--tlsv1.2
--secure-protocol=TLSv1_3	--tlsv1.3
--https-only	--proto =https
--ciphers=str	--ciphers <list>
--no-check-certificate	-k, --insecure
--certificate=file	-E <certificate>, --cert <certificate>
--certificate-type=type	--cert-type <type>
--private-key=file	--key <key>
--private-key-type=type	--key-type <type>
--ca-certificate=file	--cacert <file>
--ca-directory=directory	--capath <dir>
--crl-file=file	--crlfile <file>
--pinnedpubkey=file/hashes	--pinnedpubkey <hashes>
--random-file=file	--random-file <file>
--egd-file=file	--egd-file <file>
--no-hsts	(do not set the --hsts option)
--hsts-file=file	--hsts <filename> (default '~/.wget-hsts', but incompatible format)
--ftp-user=user --ftp-password=password	(add user and password to any FTP URLs)
--no-glob	-g, --globoff
--no-passive-ftp	-P -, --ftp-port -
--ftps-implicit	--ssl-reqd
--ftps-clear-data-connection	--ftp-ssl-control
--ftps-fallback-to-ftp	--ssl

Note that this does not include any of the --no-* options that are not explicitly documented. Wget2 also added some new options, which are not yet included. Some of these options require a specific version of cURL, which would need to be checked. Please let me know if you find any errors in this table, as in some cases the documentation was unclear.

Wget options unsupported by cURL (click to show)

These options should output an error if used.

• -b, --background
• -e command, --execute command
• -o logfile, --output-file=logfile
• -a logfile, --append-output=logfile
• -d, --debug
• --report-speed=type
• --keep-badhash
• --metalink-over-http
• --metalink-index=number
• --preferred-location
• -F, --force-html
• -B URL, --base=URL
• --config=FILE
• --rejected-log=logfile
• --bind-address=ADDRESS
• --backups=backups
• --progress=type
• --no-if-modified-since
• --dns-timeout=seconds
• --read-timeout=seconds
• -w seconds, --wait=seconds
• --random-wait
• -Q quota, --quota=quota
• --no-dns-cache
• --restrict-file-names=modes
• --prefer-family=none/IPv4/IPv6
• --use-askpass=command
• --no-iri
• --local-encoding=encoding
• --unlink
• -nd, --no-directories
• -nH, --no-host-directories
• --protocol-directories
• --cut-dirs=number
• -E, --adjust-extension
• --no-cache
• --trust-server-names
• --retry-on-host-error
• --retry-on-http-error=code[,code,...]
• --secure-protocol=PFS
• --warc-file=file
• --warc-header=string
• --warc-max-size=size
• --warc-cdx
• --warc-dedup=file
• --no-warc-compression
• --no-warc-digests
• --no-warc-keep-log
• --warc-tempdir=dir
• --no-remove-listing
• --preserve-permissions
• --retr-symlinks
• --no-ftps-resume-ssl
• -r, --recursive
• -l depth, --level=depth
• --delete-after
• -k, --convert-links
• --convert-file-only
• -K, --backup-converted
• -m, --mirror
• -p, --page-requisites
• --strict-comments
• -A acclist, --accept acclist
• -R rejlist, --reject rejlist
• --accept-regex urlregex
• --reject-regex urlregex
• --regex-type regextype
• -D domain-list, --domains=domain-list
• --exclude-domains domain-list
• --follow-ftp
• --follow-tags=list
• --ignore-tags=list
• --ignore-case
• -H, --span-hosts
• -L, --relative
• -I list, --include-directories=list
• -X list, --exclude-directories=list
• -np, --no-parent

cURL options needed to match Wget defaults

Some of these options may need to be removed based on usage of the above options.

• --retry 20
• -n, --netrc (or --netrc-optional)
• --no-clobber
• -R, --remote-time
• --retry-max-time 10
• -b '', --cookie ''
• -L, --location
• --max-redirs 20
• -f, --fail
• --remote-name-all (does not support percent decoding)
• --hsts '' (in-memory cache or create an equivalent of '~/.wget-hsts')

Current wcurl default options:

wcurl/wcurl

Lines 109 to 114 in 800c41c

	--fail \
	--globoff \
	--location \
	--proto-default https \
	--remote-time \
	--retry 5 "

wcurl/wcurl

Lines 191 to 192 in 800c41c

	# --no-clobber is only supported since 7.83.0.
	# --parallel is only supported since 7.66.0.

Also see #55.

[samueloph]

wcurl does not aim at being a full replacement of wget, the main goal is to just download files without having to remember parameters.

If there's an option which would benefit a lot of users and that it could be implemented in POSIX shell without too much added complexity, then we can consider it, but each option would have to be thoroughly discussed.

After all, if you need to start hunting down for unusual parameters for your transfer, you might as well use curl directly.

wcurl provides the --curl-options parameter which lets you set any of curl's parameters, and wcurl --dry-run example.com can be used as a starting point if you want to invoke curl directly, as it prints out what would be the curl invocation for that transfer.

[tdulcet]

Sure, but as I explained above, wcurl is being shipped as a replacement for wget by some distributions, but yet it currently only supports a single one of wget's options (-O). The point is to avoid breaking scripts and command line invocations when possible where the user was expecting wget, but it is actually wcurl. Of course, as noted above, curl/wcurl would not be able to replace all of wget's functionality (it would need a HTML and CSS parser), but it should be able to replace a majority if wgets existing usage of simply downloading files. Based on the data above, at least 82 of wget's 160 options could be supported by wcurl, or 51.25%.

Supporting most of those 82 wget options would simply require translating from the wget command line option to the equivalent for curl, as listed in the table. For example, if the user provides the --no-clobber option, it would need to be translated to --skip-existing. Several of the command line options, such as --xattr, could be left unchanged. There are also a few default options that would need to be changed to match wget, for example, --hsts would need to be provided and --retry 20 would need to be used instead of --retry 5. It would of course be easier to do this with the Bash shell and getopt, but this could also be done with a POSIX shell, as wcurl already does.