Everything work just fine, except a crawl order, I add a priority method but didn`t work correctly.
Need to first write all author data, then all album and songs data and store to DB with this order.
I want to query items in a MySQL table by order from item in another one and make a better performance. On other words - need to rewrite some part of item pipeline (process_item method) to this work correctly.
First write all author items in Author table, and then order album items in Album table by authorId from Author table.
Thanks write other tables (what works) and go to another pages.
P.S. I didn`t know what missed, maybe Spider code need to be refactorized for different order.
Always is issue with a select query from another table and comparison like on example below.
(SELECT AuthorId FROM author WHERE AuthorName ='" + str([login to view URL]("albumAuthor")) + "')
Also I need to remove a duplicated items from a some attributes. Have a three item class for author, album and song parser. I was tried to make a another flow of spider and put all in one item class, but with no success. Order was a same. :(
Database structure: [login to view URL]
Github repository: [login to view URL]
13 freelancers are bidding on average €15/hour for this job
Hello. I'm a full-stack and full-time developer. I'm an expert of scraping. I have scrapped many sites. Look at my portfolio. If you want hire me, I will do my best for your project. I hope many discussions. Thanks.