v0.3.1

v0.3.0 kept search working and put the search page behind --no-robots. That was the right call about permission and the wrong one about data: the record it handed back carried four fields per row and threw the rest away. This release reads both search surfaces for everything they carry and folds them into the graph.

suggest, under its own name

/book/auto_complete is the search route robots.txt allows, and it now has a command of its own:

goodread suggest "hunger games"

It gets a name because "search without --deep" describes a flag rather than a thing, and a script that wants the endpoint needing no override should be able to ask for it.

It is also the only surface on the site that hands over a book id and its work id in the same response. Getting from an edition to its work normally costs a request, so every row now writes an edition_of edge for free. The rest of the row lands too: the title with and without its series, the series and position parsed out of the suffix, the cover, the author with their profile and their works list, the average rating, the ratings count, the page count, the description with and without markup, whether that description was truncated and where the rest of it is, and the tracking the endpoint attaches to its own links.

It publishes no total and it does not paginate, and the record says so every time rather than looking like it returned everything.

The search page, in full

goodread search "the hunger games" --deep --no-robots --pages 2

The page is worth the flag for what surrounds the rows as much as for the rows. It states the range and the total, hedged the way the site hedges it, how long the site says it took, the qid that ties the ranks to one run, the genre it mapped the query to, eleven related shelves with their site wide counts, and how far the pager goes. Each row carries its rank, the book, the work id off the editions link, the edition count, the author, the cover, the average rating and count, the year, and both the clean link and the tracked one the page wrote.

Pages beyond the first are followed through the site's own next link, because that link carries the qid and a walk that dropped it would be a fresh search per page with ranks that do not line up. --limit trims the rows and nothing else: the totals stay as the site stated them.

The rows are read by the same parser as the author and Listopia pages, because the markup is the same markup.

The tabs that do not answer

Three of the five tabs, Listopia, groups and quotes, reply to an anonymous reader with an AWS WAF challenge.

--type lists, --type groups and --type quotes refuse before a request is spent and exit 4, and the refusal says what the site does rather than pretending the tool cannot. There is no cookie flag, no browser and no code path that tries to satisfy a challenge, and a test asserts that those three tabs cost zero requests. --no-robots does not change any of this: a published rule you decide to step over is not the same thing as an access control somebody put in your way.

--type people answers with zero rows when nobody is signed in, and the record reports that as the site's answer rather than as a parse failure.

similar says what it is

/book/similar/<id> was the recommendation surface and it no longer answers, and the "Readers also enjoyed" strip is not in the HTML an anonymous reader gets.

What similar returns is the other books the page links to, which is usually the rest of the series and the books the reviews mention. The command stays, because a link list is still useful for crawling, and the help now says plainly that these are not recommendations and are not ranked.

Also

Listing rows now write the author as a node and not only as an edge, so a search or a list read leaves authors you can actually reach. --store still fills only through crawl, bulk and lookup, which is unchanged.