Don’t Keep Requesting API Data, DO THIS!
When working with APIs, it’s important to optimize your requests to minimize load on the server and reduce latency. One common mistake that developers make is repeatedly requesting the same data from an API, which can lead to unnecessary strain on the server and slower performance for your application.
Instead of making repeated API requests, consider implementing a caching strategy to store the data locally and only make a new request when the data has changed. This can significantly improve the performance of your application and reduce the load on the API server.
Use Local Storage
One way to cache API data is to store it in the browser’s local storage. This allows you to retrieve the data quickly without making a request to the server every time. However, keep in mind that local storage has limitations in terms of the amount of data that can be stored, so be sure to only cache the data that is necessary for your application.
Implement Caching on the Server Side
Another approach to caching API data is to implement server-side caching. This involves storing the API response in a cache on the server, such as in memory or in a database, and serving the cached data to subsequent requests. This can help reduce the load on the API server and improve response times for your application.
Utilize ETags
ETags (entity tags) are another tool that can be used to optimize API requests. An ETag is a unique identifier that is associated with a specific version of a resource. When making a request to an API, you can include the ETag in the request headers, and the server will only send the data if the ETag has changed. This can help reduce unnecessary data transfer and improve performance.
Conclusion
By implementing a caching strategy and optimizing your API requests, you can improve the performance of your application and reduce the load on the API server. Consider using local storage, server-side caching, and ETags to minimize repeated requests and improve the efficiency of your application.
Grab IPRoyal Proxies with a 30% discount while it lasts! Click here to find the 30% off coupon code: 02:05 Find out more at iproyal.club/JohnWatsonRooney
does this work with httpx? Been getting some errors
I just used JWR50 for two different orders and it still works, hopefully you got the payments for it.
A little code update is required to make the code from this tutorial run.
requests_cache released version 1.0.0 on 2023-03-01 with a breaking change:
line 23 and 26 now need () after session.cache.urls as it has been replaced with a method that returns a list of sorted unique URLs.
Is there a way to cache or log your POST requests to an endpoint? A use case could be, I want to know what endpoint I hit that received a 500 response, so I can try again at a later time.
hey John. I've been watching a lot of. your videos lately – I'm trying to scrape agent data from recently sold listings (given an address, navigate to the page, grab agent name, phone number, name, and email
I'm able to get the info I want, but after a couple requests (never more than 3), Zillow hits me with a Press and Hold captcha that I can't get past.
I've tried using paid proxies, requests, beautifulsoup, playwright, headless=false, hitting the zillow homepage and entering in the search terms instead of navigating directly to the url…. haven't been able to figure out how zillow knows its me.
Any suggestions? Could potentially make for a good video as I'm sure other sites employ similar tactics. My next attempt is going to be to use your back end scraping method.
Thanks man!
could you share you vim set up ? 🙂
Which distro and window manager are you using?
Thank so much,
When can I learn advanced things like these to improve my web scraping level ?
Hi, Can you make a video about your Vim editor configuration?
I was using jupyter notebook (in vscode) this whole time.
Very good, what are the recommended sites for learning crawlers?
thank you for the video, how about fast api with redis cache ?
Good stuff again !!
API caching 🥳 Interesting topic, thank you John!
And in 4K! 🤩🙏
Thanks for this kinda useful stuff.
What editor are you using?
thanks boss . Can you explain how to confirm temp emails when registering on a site for the first time? please
Yesss always trying to learn more FastAPI