Thursday, April 25, 2024
 Popular · Latest · Hot · Upcoming
67
rated 0 times [  67] [ 0]  / answers: 1 / hits: 139363  / 3 Years ago, thu, june 17, 2021, 12:39:35

Some parts of wikipedia appear differently when you're logged in. I would like to wget user pages so they would appear as if I was logged in.



Is there a way I can wget user pages like this



http://en.wikipedia.org/wiki/User:A


this is the login page:



http://en.wikipedia.org/w/index.php?title=Special:UserLogin&returnto=Login&campaign=ACP3

More From » bash

 Answers
4

###The easy way: login with your browser,and give the cookies to wget


Easiest method: in general, you need to provide wget or curl with the (logged-in) cookies from a particular website for them to fetch pages as if you were logged in.


If you are using Firefox, it's easy to do via the cookie.txt add-on. Install the add-on, and:



  1. Click on the plugin and save the cookies.txt file (you can change the filename/destination).



  2. Open up a terminal, and use wget with the --load-cookies=FILENAME option, e.g.


     wget --load-cookies=cookies.txt http://en.wikipedia.org/wiki/User:A




  • For curl, it's curl --cookie cookies.txt ...


(I will try to update this answer for Chrome/Chromium users)


###The hard way: use curl (preferably) or wget to manage the entire session



  • A detailed how-to is beyond the scope of this answer, but you use curl with the --cookie-jar or wget with the --save-cookies --keep-session-cookiesoptions, along with the HTTP/S PUT method to log in to a site, save the login cookies, and then use them to simulate a browser.

  • Needless to say, this requires going through the HTML source for the login page (get input field names, etc.), and is often difficult to get to work for sites using anything beyond simple login/password authentication.

  • Tip: if you go this route, it is often much simpler to deal with the mobile version of a website (if available), at least for the authentication step.


[#36970] Saturday, June 19, 2021, 3 Years  [reply] [flag answer]
Only authorized users can answer the question. Please sign in first, or register a free account.
herriail

Total Points: 11
Total Questions: 122
Total Answers: 118

Location: Peru
Member since Tue, Mar 16, 2021
3 Years ago
herriail questions
Tue, May 18, 21, 02:11, 3 Years ago
Sun, Dec 12, 21, 06:06, 2 Years ago
Wed, May 24, 23, 01:42, 1 Year ago
Sun, Jan 29, 23, 00:48, 1 Year ago
;