Can AI Make Recommendations Safer?
Recommendation engines decide what a shopper sees first, and the data that makes a suggestion useful can also make pressure precise. Australian surveys find manipulative online design is common, and a ban on unfair trading practices takes effect in July 2027. Whether AI recommendations are safe depends on the limits a retailer puts on its own optimisation, and the evidence that those limits work is still thin.
On this page
Better: Note and confidence
Low confidenceUseful suggestions; the benefit is measured in sales, not shopper outcomesPublished retail field tests show modest revenue gains, and the large headline figures are company statements. We found no independent study measuring whether shoppers end up with better purchases.Faster: Note and confidence
Low confidenceShorter search; pressure works faster tooNarrowing a large catalogue saves effort, but we found no independent measure of the time it saves shoppers. Countdown timers and scarcity cues exist to speed decisions up, which is not always in the shopper's interest.Cheaper: Note and confidence
Low confidenceRevenue for the seller; some of the cost lands on the shopperRecommendations add sales in published trials. One in five Australians surveyed by the CPRC had spent more than they intended because of online design, so in our reading part of a seller's gain can be a shopper's cost.Safer: Note and confidence
Governance-gatedSafe only when the optimisation has limitsThe data that personalises a suggestion can personalise pressure. Europe requires recommender transparency, and a non-profiling option on the largest platforms; Australia's ban on unfair trading practices starts on 1 July 2027. We found no study showing these guardrails reduce harm.The problem, in numbers
A recommendation is a design choice. It decides which product appears first, which option comes pre-ticked and which reminder arrives after a shopper leaves a cart. When the choice serves the shopper, it saves effort. When it steers shoppers into choices they would not otherwise make, regulators call it a dark pattern.
Many Australians report harm from these patterns. In a nationally representative survey of 2,000 people in April 2022, the Consumer Policy Research Centre found 83% had experienced at least one negative consequence from website or app design aimed at influencing behaviour. One in five had spent more than they intended, one in four had shared more personal information than they wanted to, and nearly one in ten had bought something by accident.Source 1 The ACCC's own survey, run in late 2024, found that 72% of respondents who had bought on a general online retail marketplace in the previous year had met a potentially unfair practice, such as hidden charges or accidental clicks.Source 2
International sweeps point the same way. In 2024, 27 consumer agencies from 26 countries examined 642 traders selling subscriptions: 75.7% used at least one dark pattern and 66.8% used several. Where subscriptions renewed automatically, 81% gave no way to switch renewal off during the purchase.Source 4 A parallel privacy sweep of more than 1,000 sites and apps found 57% made the least privacy-protective option the most obvious.Source 4
What improves, and why
In our reading, retailers build recommendation engines because they sell. A peer-reviewed review of recommender field tests collects the best-known company figures: Netflix estimated the value of recommendation and personalisation at more than US$1 billion a year, and Amazon's chief executive was reported in 2006 as saying about 35% of sales came from cross-sales. The reviewers note that how the Netflix figure was estimated is not specified.Source 8 These are company statements, not independent measurements. Published retail trials show smaller effects: revenue from recommendation lists rose 1.8% in one grocery pilot and direct revenue 0.3% in another, while an online bookstore lost 17% of revenue when its recommendations were switched off, though the reviewers say seasonal effects may have contributed.Source 8
For shoppers, a good recommendation narrows a large catalogue to things they are likely to want. In our reading, that is a real benefit, but the field tests in that review measure it in sales and clicks, not in whether shoppers are better off. The same review warns that click-through rates can mislead, and that almost none of the tests it surveyed reported sample-size analysis or detailed statistics.Source 8
In our reading, the safety problem sits in the optimisation. The FTC's staff report on dark patterns notes that online sellers can test designs "more easily, frequently, and at a much larger scale" than physical stores. It describes a case in which, according to the FTC's complaint, A/B testing showed a false "pre-approved" claim drew more clicks, and the company chose it.Source 7 Treasury's 2024 consultation draws the link to personal data: businesses that hold information about a consumer can target them with personalised messages, notifications and suggestions.Source 3 The ACCC adds that platforms can tailor the products and services they show using demographic data and user behaviour.Source 2 An engine tuned only for conversion will, on that reading, learn whatever converts, including pressure.
A recommender tuned for conversion cannot tell a helpful suggestion from a well-timed push. Someone has to tell it where the line is.
What the evidence actually says
Manipulative design changes behaviour, and strongly. In a peer-reviewed US experiment, 1,963 people were offered a data and identity-theft protection plan. With a neutral interface, 11.3% accepted. Mild dark patterns more than doubled that, to 25.8%, and aggressive ones took it to 41.9%. Less educated participants were more likely to accept when dark patterns were used.Source 7 The setting is not retail, but in our reading the mechanics are the ones retail interfaces use.
Study 1 of Luguri and Strahilevitz, census-weighted US sample of 1,963 participants. Bar widths are our calculation, scaled to the largest value.
On shopping sites themselves, a peer-reviewed 2019 crawl of about 11,000 shopping websites and 53,000 product pages found 1,818 instances of dark patterns across 15 types, 183 websites engaging in deceptive practices, and 22 third-party firms offering such patterns as a ready-made service.Source 4 In our reading, that last finding matters for AI: pressure can arrive as a plug-in, not only as an in-house design.
What we did not find is evidence that guardrails on recommendation engines reduce harm. No independent study we found measures whether a non-profiling option, a transparency statement or a design review leaves shoppers better off. Attitudes, at least, are clear. In the OAIC's 2023 survey of 1,916 Australian adults, 69% said tracking, profiling and targeted advertising based on personal information is not fair and reasonable, yet 53% would rather see targeted ads if they must see ads at all.Source 5 In our reading, shoppers accept relevance; they object to being profiled to get it.
Where it falls short
Measured for the seller. Field tests count revenue and clicks, and most lack detailed statistics.Source 8 We found no independent study of whether shoppers end up better off.
Patterns stack. Dark patterns tend to work harder in combination,Source 7 and two-thirds of subscription sellers swept used more than one.Source 4
Pressure can be bought in. Some third-party firms offer dark patterns as a turnkey product,Source 4 so a retailer can install one without designing it.
Where governance decides the outcome
The rules are arriving. Europe's Digital Services Act bars online platforms from interfaces that deceive or manipulate users or materially distort their ability to make free and informed decisions. It requires platforms to explain the main parameters of their recommender systems in plain language, and very large platforms, with more than 45 million monthly EU users, must offer at least one recommender option not based on profiling, with fines of up to 6% of worldwide turnover.Source 6 In Australia, Parliament passed the unfair trading practices bill in 2026 and the ban commences on 1 July 2027.Source 3 Treasury's consultation proposed that it reach conduct that unreasonably distorts or manipulates a consumer's decisions and causes or is likely to cause material detriment, including the unreasonable use of dark patterns.Source 3 A retailer using AI recommendations should name an owner for the objective the engine optimises, review any test that touches defaults, urgency or price before it ships, check third-party plug-ins for pressure tactics, and offer shoppers a version that is not built on their profile.
How to think about adoption
Start by writing down what the recommendation engine is rewarded for. If the only target is conversion, add measures that capture regret: returns, cancellations, complaints and accidental sign-ups. Audit the checkout and subscription flows against the patterns the regulators swept for, such as pre-ticked boxes, hidden renewal terms and fake urgency, practices that both the international sweeps and Australia's reform debate have singled out. Treat vendor uplift claims as hypotheses and test them against a simple baseline, with enough shoppers and enough weeks to see longer-term effects. And give customers a plain explanation of why they see what they see. A recommendation that a shopper would accept with the reasons in view is one a retailer can defend.
BFCS.ai does not fabricate figures. Company figures on the value of recommendations are self-reported and reach us second-hand through a peer-reviewed review; we label them as such. The strongest evidence here is about harm from manipulative design. We found no independent study measuring whether guardrails on recommendation engines reduce that harm, and we say so where it matters.
Sources
- 1
Consumer Policy Research Centre (CPRC), "Duped by Design: Manipulative online design: Dark patterns in Australia," 8 June 2022, and media release "New protections needed to stop online designs that hurt consumers." CPRC surveyed 2,000 Australians in April 2022; the survey is described as nationally representative. cprc.org.au
Back to textWhat we took from it
83% had experienced one or more negative consequences from websites or apps using design features aimed at influencing behaviour; 20% spent more than they intended; 25% shared more personal information than they wanted to; 9% accidentally bought something. CPRC also ran a randomised sweep of websites and apps; the pages do not state how many were swept.
- 2
Australian Competition and Consumer Commission, "Digital platform services inquiry: Final report," March 2025. accc.gov.au
Back to textWhat we took from it
ACCC consumer survey data (Lonergan Research, survey of Australian consumers aged 14+, October to November 2024): 72% of respondents who had used a general online retail marketplace to make a purchase in the previous 12 months had encountered at least one potentially unfair practice in that time, such as hidden charges or accidental clicks (Figure E.3, p 5). The report, quoting other work, says algorithms can change how choice is personalised and presented, and that platforms can tailor the content served to users, including products and services, based on demographic data and user behaviour (p 60). Recommendation 1 calls for an economy-wide prohibition against unfair trading practices. The ACCC website refused automated access (403, tried twice on 31 October 2026); the text was read from a third-party copy hosted by PPC Land, which carries the ACCC cover, page headers and Commonwealth copyright notice (CC BY 4.0) and which we have not compared with the ACCC original.
- 3
The Treasury (Australian Government), "Unfair trading practices: Consultation on the design of proposed general and specific prohibitions," November 2024 (submissions closed 13 December 2024). treasury.gov.au
CPRC, "Countdown is on until unfair is illegal," media release, 2 July 2026: cprc.org.au
Assistant Minister for Productivity, Competition, Charities and Treasury, "Unfair trading tricks and traps to be banned," media release, 2 July 2026: ministers.treasury.gov.au
Back to textWhat we took from it
Proposes a general prohibition capturing conduct that "unreasonably distorts or manipulates, or is likely to unreasonably distort or manipulate," a consumer's economic decision-making or behaviour and causes, or is likely to cause, material detriment, including the unreasonable use of dark patterns; notes that where businesses hold personal information they can target consumers with personalised messaging, notifications, suggestions and other design adjustments; describes countdown timers and low-stock notifications used to create urgency; and reproduces Article 25(1) of the EU Digital Services Act.
the Competition and Consumer Amendment (Unfair Trading Practices) Bill 2026 has passed Parliament, introducing an economy-wide ban on unfair trading practices and measures on subscription traps and drip pricing, set to commence on 1 July 2027.
the Bill passed Parliament that day; unfair trading practices and subscription traps are banned from 1 July 2027, after a 12-month lead-in.
- 4
International Consumer Protection and Enforcement Network (ICPEN), "Dark Patterns in Subscription Services Sweep: Public Report," 2 July 2024. icpen.org
Office of the Australian Information Commissioner, "GPEN Sweep finds majority of websites and mobile apps use deceptive design to influence privacy choices," 10 July 2024: oaic.gov.au
Mathur, A., Acar, G., Friedman, M. J., Lucherini, E., Mayer, J., Chetty, M. & Narayanan, A., "Dark Patterns at Scale: Findings from a Crawl of 11K Shopping Websites," Proceedings of the ACM on Human-Computer Interaction 3 (CSCW), article 81, November 2019 (peer-reviewed). arxiv.org
Back to textWhat we took from it
27 consumer protection agencies from 26 countries, run alongside the GPEN privacy sweep; the report gives no sweep dates; of 642 traders selling subscriptions, 75.70% used at least one dark pattern and 66.82% used several; 81% of traders with automatically renewing subscriptions gave no way to turn off auto-renewal in the purchase flow.
26 privacy enforcement authorities examined more than 1,000 websites and apps from 29 January to 2 February 2024; 57% made the least privacy-protective option the most obvious; the sweep was not an investigation.
Abstract: about 11K shopping websites and 53K product pages; 1,818 dark pattern instances of 15 types in 7 categories; 183 websites engaging in deceptive practices; 22 third-party entities offering dark patterns as a turnkey solution. Figures are from the abstract of the author version.
- 5
Office of the Australian Information Commissioner, "Australian Community Attitudes to Privacy Survey 2023," published 8 August 2023 (Lonergan Research, March 2023; nationally representative sample of 1,916 adults). oaic.gov.au
Back to textWhat we took from it
69% say online tracking, profiling and targeted advertising based on personal (but not sensitive) information is not fair and reasonable, against 24% who say it is; if they must receive ads, 53% would prefer them targeted and relevant, up from 48% in 2020; only 32% feel in control of their data privacy.
- 6
Regulation (EU) 2022/2065 (Digital Services Act). springlex.eu
European Commission, "The Digital Services Act package": digital-strategy.ec.europa.eu
Back to textWhat we took from it
Article 25(1): providers of online platforms shall not design, organise or operate their interfaces in a way that deceives or manipulates users or otherwise materially distorts or impairs their ability to make free and informed decisions (as reproduced in the Treasury paper, note 3). Article 27: platforms using recommender systems must set out in their terms and conditions, in plain and intelligible language the main parameters used and any options for users to modify or influence them. Article 38: very large online platforms and search engines must provide at least one option for each recommender system that is not based on profiling. Article 74(1): fines on very large platforms of up to 6% of total worldwide annual turnover. Articles 27, 38 and 74 read on Springlex, an unofficial reproduction that says it has no legal effect and that the Official Journal version (OJ L 277, 27 October 2022) is authentic; the EUR-Lex page was tried again on 31 October 2026 and returned no readable text.
very large online platforms are those with over 45 million monthly users in the EU, and users of large platforms can opt for non-personalised feeds.
- 7
US Federal Trade Commission, Bureau of Consumer Protection, "Bringing Dark Patterns to Light," staff report, September 2022. ftc.gov
Luguri, J. & Strahilevitz, L. J., "Shining a Light on Dark Patterns," Journal of Legal Analysis 13(1), 2021, pp. 43-109 (peer-reviewed). academic.oup.com
Back to textWhat we took from it
Companies that market online can experiment with dark patterns "more easily, frequently, and at a much larger scale" than traditional retailers; describes the FTC's action against Credit Karma, which, according to the FTC's complaint, used A/B testing and chose a false "pre-approved" claim because it yielded a greater click rate; dark patterns tend to have stronger effects when combined.
Study 1: census-weighted US sample of 1,963 participants offered an identity-theft protection plan; acceptance was 11.3% in the control condition, 25.8% with mild dark patterns and 41.9% with aggressive dark patterns; less educated participants were more likely to accept, an effect significant in the dark pattern conditions but not the control; with income controlled, it held for mild but not aggressive dark patterns. Bar widths in the figure are BFCS.ai calculations scaled to the largest value.
- 8
Jannach, D. & Jugovac, M., "Measuring the Business Value of Recommender Systems," ACM Transactions on Management Information Systems 10, 2019, doi:10.1145/3370082 (peer-reviewed journal; this article relies on the authors' arXiv version, v3, 17 December 2019). arxiv.org
Back to textWhat we took from it
Second-hand company figures: Netflix estimated the value of recommendation and personalisation at more than US$1 billion a year (Gomez-Uribe and Hunt, 2015), and the authors note how this was estimated is not specified; about 35% of Amazon's sales were reported to originate from cross-sales, citing a 2006 statement by Amazon's chief executive in online media. Field tests: grocery revenue up 1.8% through purchases made directly from recommendation lists, in a pilot (Lawrence et al., 2001); direct revenue up 0.3% in another grocery test (Dias et al., 2008); an online bookstore's revenue fell 17% when recommendations were removed for a month (Shani et al., 2005); the reviewers say seasonal effects could also have influenced this, and their summary describes the removal as lasting a week. The Netflix figure comes from a paper by Netflix staff. The authors warn that click-through rates can be misleading and that sample-size analyses and detailed statistics were missing in almost all surveyed cases.