[kwlug-disc] Utility to parse HTTP response

[William Park]

Maybe someone already wrote a standalone utility based on Python HTTP 
libraries.  Thanks.


On 2025-01-31 09:15, Khalid Baheyeldin wrote:
> On Fri, Jan 31, 2025 at 1:16 AM William Park via kwlug-disc <kwlug- 
> disc at kwlug.org <mailto:kwlug-disc at kwlug.org>> wrote:
> 
>     Hi all, (also posted to GTALUG)
> 
>     To build HTTP request, you use 'curl'.
> 
>     After you get HTTP response, what utility do you use to extract stuffs
>     you want?  JSON format has 'jq', and XML format has 'xmlstarlet'.  I'm
>     looking for something like that but for HTTP format.
> 
>     I know it's simple (HTTP response has 3 parts: first line, headers, and
>     body), but I just want to reduce chance of typos.
> 
> 
> Do you want to do this from the bash, in shell scripts?
> Or something else.
> 
>  From shell scripts, I do this:
> 
> curl -s -k -L -i -o $TMP -H "User-Agent: $UA" "$URL" 2>&1
> 
> # Print the headers
> sed -ne '1,/^^M$/p' $TMP
> 
> That ^M is actually a Ctrl-M which separates the header portion
> from the returned content.
> 
> For kwlug.org <http://kwlug.org> for example, it returns this
> 
> HTTP/1.1 301 Moved Permanently
> Date: Fri, 31 Jan 2025 14:04:01 GMT
> Server: Apache
> X-Content-Type-Options: nosniff
> Location: http://kwlug.org/ <http://kwlug.org/>
> Cache-Control: max-age=31536000
> Expires: Sat, 31 Jan 2026 14:04:01 GMT
> Content-Length: 225
> Content-Type: text/html; charset=iso-8859-1
> 
> That of course means the headers and the content are merged in
> one output file, and you have to extract what you need with lots
> of code.
> 
> If it is something more complex, then I use Python's requests library.
> You need to use exceptions to handle things like timeouts and too
> many redirects, ...etc.
> 
> Abrief tutorial can be found here (and elsewhere)
> 
> https://www.geeksforgeeks.org/python-requests-tutorial/ <https:// 
> www.geeksforgeeks.org/python-requests-tutorial/>
> 
> -- 
> Khalid M. Baheyeldin
> 
> 
> _______________________________________________
> kwlug-disc mailing list
> To unsubscribe, send an email to kwlug-disc-leave at kwlug.org
> with the subject "unsubscribe", or email
> kwlug-disc-owner at kwlug.org to contact a human being.